# BERT Question Answering in TensorFlow with Mixed Precision Copyright 2021 NVIDIA Corporation. All Rights Reserved. Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at http://www.apache.org/licenses/LICENSE-2.0 Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License. ```python !nvidia-smi ``` Tue Mar 16 00:26:48 2021 +-----------------------------------------------------------------------------+ | NVIDIA-SMI 460.32.03 Driver Version: 460.32.03 CUDA Version: 11.2 | |-------------------------------+----------------------+----------------------+ | GPU Name Persistence-M| Bus-Id Disp.A | Volatile Uncorr. ECC | | Fan Temp Perf Pwr:Usage/Cap| Memory-Usage | GPU-Util Compute M. | | | | MIG M. | |===============================+======================+======================| | 0 Tesla V100-DGXS... On | 00000000:08:00.0 Off | 0 | | N/A 37C P0 39W / 300W | 5MiB / 32508MiB | 0% Default | | | | N/A | +-------------------------------+----------------------+----------------------+ +-----------------------------------------------------------------------------+ | Processes: | | GPU GI CI PID Type Process name GPU Memory | | ID ID Usage | |=============================================================================| +-----------------------------------------------------------------------------+ ## 1. Overview Bidirectional Embedding Representations from Transformers (BERT), is a method of pre-training language representations which obtains state-of-the-art results on a wide array of Natural Language Processing (NLP) tasks. The original paper can be found here: https://arxiv.org/abs/1810.04805. NVIDIA's BERT is an optimized version of Google's official implementation, leveraging mixed precision arithmetic and tensor cores on V100 GPUS for faster training times while maintaining target accuracy. ### Learning objectives This notebook demonstrates: - Inference on Question Answering (QA) task with BERT Large model - The use/download of fine-tuned NVIDIA BERT models from [NGC](https://ngc.nvidia.com) - Use of Mixed Precision models for Inference ## 2. Setup ### Pre-Trained NVIDIA BERT TensorFlow Models on NGC

We will be using the following configuration of BERT in this example: | **Model** | **Hidden layers** | **Hidden unit size** | **Attention heads** | **Feedforward filter size** | **Max sequence length** | **Parameters** | |:---------:|:----------:|:----:|:---:|:--------:|:---:|:----:| |BERTLARGE|24 encoder|1024| 16|4 x 1024|512|330M| **To do so, we will take advantage of the pre-trained models available on the [NGC Model Registry](https://ngc.nvidia.com/catalog/models).** Among the many configurations available we will download one of these two: - **bert_tf_ckpt_large_qa_squad2_amp_384** which are trained on the [SQuaD 2.0 Dataset](https://rajpurkar.github.io/SQuAD-explorer/). We can choose the mixed precision model (which takes much less time to train than the fp32 version) without losing accuracy, with the following flag: ```python use_mixed_precision_model = True ``` ```python # bert_tf_ckpt_large_qa_squad2_amp_384 DATA_DIR_FT = '/workspace/bert/data/finetuned_large_model_SQUAD2.0' !mkdir -p $DATA_DIR_FT !wget --content-disposition -O $DATA_DIR_FT/bert_tf_ckpt_large_qa_squad2_amp_384_19.03.1.zip \ https://api.ngc.nvidia.com/v2/models/nvidia/bert_tf_ckpt_large_qa_squad2_amp_384/versions/19.03.1/zip \ && unzip -n -d $DATA_DIR_FT/ $DATA_DIR_FT/bert_tf_ckpt_large_qa_squad2_amp_384_19.03.1.zip \ && rm -rf $DATA_DIR_FT/bert_tf_ckpt_large_qa_squad2_amp_384_19.03.1.zip ``` --2021-03-16 00:19:46-- https://api.ngc.nvidia.com/v2/models/nvidia/bert_tf_ckpt_large_qa_squad2_amp_384/versions/19.03.1/zip Resolving api.ngc.nvidia.com (api.ngc.nvidia.com)... 52.9.83.149, 52.9.91.161 Connecting to api.ngc.nvidia.com (api.ngc.nvidia.com)|52.9.83.149|:443... connected. HTTP request sent, awaiting response... 302 Location: https://s3.us-west-2.amazonaws.com/prod-model-registry-ngc-bucket/org/nvidia/models/bert_tf_ckpt_large_qa_squad2_amp_384/versions/19.03.1/files.zip?response-content-disposition=attachment%3B%20filename%3D%22files.zip%22&response-content-type=application%2Fzip&X-Amz-Security-Token=IQoJb3JpZ2luX2VjEM%2F%2F%2F%2F%2F%2F%2F%2F%2F%2F%2FwEaCXVzLXdlc3QtMSJIMEYCIQCrGyjQ8LQkZNyTMr6nchH4edRFx%2F25OXI%2FNE2G%2BBF8SQIhAKduF2AEyjRXyw4e5%2BsfnhA81SY%2BN4DOACIaawtQOboyKr0DCPj%2F%2F%2F%2F%2F%2F%2F%2F%2F%2FwEQAxoMNzg5MzYzMTM1MDI3IgxYmy9jrsgCYn3zEQkqkQOLHUCtPKMUjTXYRa2arifO0nTVXmCxHLV%2FCbZmmZOF5J1HGHQgnyZu1y3EPoh8cf5Kn5VfzORhYQlq3KvbKN5%2FXJq0170myZrAOjqrSRUE1DstQl8P0ym3V1S9EWnjYurXuNMa%2FQaqzzZLsVE%2FhuB7N2x0dlI4EZmZ%2B6fnqUlltvMqmc6p1Uh0FQnz6dQ9L1LTTTg2nPx6OAbcbLHjalsPB%2FFBNOSoVRtphUkh4d6iM9uxByZYiYAVr23aJExAU0GAJtjexOZ8bPLzgWcnDRTBzMk0LwKlOtWePK3GFFJQUMD4GOLO3wVXsXv6LHBRjOS7icqrDm7zdHS0OeUD5usHQPxwwbg0zId%2BgJTKmz6W2S9lSfHGwwv7ukxKraY%2BI3iV00qmq757DeYu0M%2FAEqOa9CyTTCq1TL%2FOqhND1Nm%2BrmqKCp%2FuKqTzz%2BzMEZNUvgA5CTAT7D4M8cg2AL0zlREf8XhnX%2FOrfEwG1mrFN9WyKJavYb13m2pX5%2FellyjGv7PeZ5Q1A2czRjqlt8FdXybAkzCywb%2BCBjrqAfbuvTtMpWQy%2B5VFa2TcyJLuc4bJ2QaGinWaSAAf%2BkOkuVUjcslbodHTDSsHEjI0ezBSvNzcLJe9HUum%2Bqx9YtKrGRqnVFzwhD15D4pD0UN737M8UJk0Zoto%2BQz0ytkhi6ST1SKhu6cJUj0iXyAZd9lpNv7MDhXgUL3dL74ngRqxbyfquO%2BmBGHbfV1%2BP%2B9V3pJqATmXNrNQ6pezbu%2FjdsMXzIO1J%2F8ThEC2JBnq%2FCPCXDNRGJ0l4zqgA35YriFsh%2BmUDvOyB6LPSfJ8DuVpiLU%2BbnyUc1s0WduA6VhlhPTaBlqUznrbMp0fRw%3D%3D&X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Date=20210316T001946Z&X-Amz-SignedHeaders=host&X-Amz-Expires=3600&X-Amz-Credential=ASIA3PSNVSIZYFQQ2SH4%2F20210316%2Fus-west-2%2Fs3%2Faws4_request&X-Amz-Signature=f987fe14d3094638bb6e6abdbe47c066e0d75ba623cf1b77cf02275f52bd8e12 [following] --2021-03-16 00:19:46-- https://s3.us-west-2.amazonaws.com/prod-model-registry-ngc-bucket/org/nvidia/models/bert_tf_ckpt_large_qa_squad2_amp_384/versions/19.03.1/files.zip?response-content-disposition=attachment%3B%20filename%3D%22files.zip%22&response-content-type=application%2Fzip&X-Amz-Security-Token=IQoJb3JpZ2luX2VjEM%2F%2F%2F%2F%2F%2F%2F%2F%2F%2F%2FwEaCXVzLXdlc3QtMSJIMEYCIQCrGyjQ8LQkZNyTMr6nchH4edRFx%2F25OXI%2FNE2G%2BBF8SQIhAKduF2AEyjRXyw4e5%2BsfnhA81SY%2BN4DOACIaawtQOboyKr0DCPj%2F%2F%2F%2F%2F%2F%2F%2F%2F%2FwEQAxoMNzg5MzYzMTM1MDI3IgxYmy9jrsgCYn3zEQkqkQOLHUCtPKMUjTXYRa2arifO0nTVXmCxHLV%2FCbZmmZOF5J1HGHQgnyZu1y3EPoh8cf5Kn5VfzORhYQlq3KvbKN5%2FXJq0170myZrAOjqrSRUE1DstQl8P0ym3V1S9EWnjYurXuNMa%2FQaqzzZLsVE%2FhuB7N2x0dlI4EZmZ%2B6fnqUlltvMqmc6p1Uh0FQnz6dQ9L1LTTTg2nPx6OAbcbLHjalsPB%2FFBNOSoVRtphUkh4d6iM9uxByZYiYAVr23aJExAU0GAJtjexOZ8bPLzgWcnDRTBzMk0LwKlOtWePK3GFFJQUMD4GOLO3wVXsXv6LHBRjOS7icqrDm7zdHS0OeUD5usHQPxwwbg0zId%2BgJTKmz6W2S9lSfHGwwv7ukxKraY%2BI3iV00qmq757DeYu0M%2FAEqOa9CyTTCq1TL%2FOqhND1Nm%2BrmqKCp%2FuKqTzz%2BzMEZNUvgA5CTAT7D4M8cg2AL0zlREf8XhnX%2FOrfEwG1mrFN9WyKJavYb13m2pX5%2FellyjGv7PeZ5Q1A2czRjqlt8FdXybAkzCywb%2BCBjrqAfbuvTtMpWQy%2B5VFa2TcyJLuc4bJ2QaGinWaSAAf%2BkOkuVUjcslbodHTDSsHEjI0ezBSvNzcLJe9HUum%2Bqx9YtKrGRqnVFzwhD15D4pD0UN737M8UJk0Zoto%2BQz0ytkhi6ST1SKhu6cJUj0iXyAZd9lpNv7MDhXgUL3dL74ngRqxbyfquO%2BmBGHbfV1%2BP%2B9V3pJqATmXNrNQ6pezbu%2FjdsMXzIO1J%2F8ThEC2JBnq%2FCPCXDNRGJ0l4zqgA35YriFsh%2BmUDvOyB6LPSfJ8DuVpiLU%2BbnyUc1s0WduA6VhlhPTaBlqUznrbMp0fRw%3D%3D&X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Date=20210316T001946Z&X-Amz-SignedHeaders=host&X-Amz-Expires=3600&X-Amz-Credential=ASIA3PSNVSIZYFQQ2SH4%2F20210316%2Fus-west-2%2Fs3%2Faws4_request&X-Amz-Signature=f987fe14d3094638bb6e6abdbe47c066e0d75ba623cf1b77cf02275f52bd8e12 Resolving s3.us-west-2.amazonaws.com (s3.us-west-2.amazonaws.com)... 52.218.250.176 Connecting to s3.us-west-2.amazonaws.com (s3.us-west-2.amazonaws.com)|52.218.250.176|:443... connected. HTTP request sent, awaiting response... 200 OK Length: 3672271344 (3.4G) [application/zip] Saving to: ‘/workspace/bert/data/finetuned_large_model_SQUAD2.0/bert_tf_ckpt_large_qa_squad2_amp_384_19.03.1.zip’ /workspace/bert/dat 100%[===================>] 3.42G 36.8MB/s in 1m 47s 2021-03-16 00:21:34 (32.6 MB/s) - ‘/workspace/bert/data/finetuned_large_model_SQUAD2.0/bert_tf_ckpt_large_qa_squad2_amp_384_19.03.1.zip’ saved [3672271344/3672271344] Archive: /workspace/bert/data/finetuned_large_model_SQUAD2.0/bert_tf_ckpt_large_qa_squad2_amp_384_19.03.1.zip inflating: /workspace/bert/data/finetuned_large_model_SQUAD2.0/bert_config.json inflating: /workspace/bert/data/finetuned_large_model_SQUAD2.0/model.ckpt.data-00000-of-00001 inflating: /workspace/bert/data/finetuned_large_model_SQUAD2.0/model.ckpt.index inflating: /workspace/bert/data/finetuned_large_model_SQUAD2.0/model.ckpt.meta inflating: /workspace/bert/data/finetuned_large_model_SQUAD2.0/tf_bert_squad_1n_fp16_gbs32.190523090758.log inflating: /workspace/bert/data/finetuned_large_model_SQUAD2.0/vocab.txt ### NGC Model Scripts While we're at it, we'll also pull down some BERT helper scripts from the [NGC Model Scripts Registry](https://ngc.nvidia.com/catalog/model-scripts/nvidia:bert_for_tensorflow) ```python # Download BERT helper scripts !wget -nc --show-progress -O bert_scripts.zip \ https://api.ngc.nvidia.com/v2/recipes/nvidia/bert_for_tensorflow/versions/1/zip !mkdir -p /workspace/bert !unzip -n -d /workspace/bert bert_scripts.zip ``` --2021-03-16 00:22:16-- https://api.ngc.nvidia.com/v2/recipes/nvidia/bert_for_tensorflow/versions/1/zip Resolving api.ngc.nvidia.com (api.ngc.nvidia.com)... 52.9.83.149, 52.9.91.161 Connecting to api.ngc.nvidia.com (api.ngc.nvidia.com)|52.9.83.149|:443... connected. HTTP request sent, awaiting response... 302 Location: https://s3.us-west-2.amazonaws.com/prod-model-registry-ngc-bucket/org/nvidia/recipes/bert_for_tensorflow/versions/1/files.zip?response-content-disposition=attachment%3B%20filename%3D%22files.zip%22&response-content-type=application%2Fzip&X-Amz-Security-Token=IQoJb3JpZ2luX2VjEM%2F%2F%2F%2F%2F%2F%2F%2F%2F%2F%2FwEaCXVzLXdlc3QtMSJIMEYCIQCrGyjQ8LQkZNyTMr6nchH4edRFx%2F25OXI%2FNE2G%2BBF8SQIhAKduF2AEyjRXyw4e5%2BsfnhA81SY%2BN4DOACIaawtQOboyKr0DCPj%2F%2F%2F%2F%2F%2F%2F%2F%2F%2FwEQAxoMNzg5MzYzMTM1MDI3IgxYmy9jrsgCYn3zEQkqkQOLHUCtPKMUjTXYRa2arifO0nTVXmCxHLV%2FCbZmmZOF5J1HGHQgnyZu1y3EPoh8cf5Kn5VfzORhYQlq3KvbKN5%2FXJq0170myZrAOjqrSRUE1DstQl8P0ym3V1S9EWnjYurXuNMa%2FQaqzzZLsVE%2FhuB7N2x0dlI4EZmZ%2B6fnqUlltvMqmc6p1Uh0FQnz6dQ9L1LTTTg2nPx6OAbcbLHjalsPB%2FFBNOSoVRtphUkh4d6iM9uxByZYiYAVr23aJExAU0GAJtjexOZ8bPLzgWcnDRTBzMk0LwKlOtWePK3GFFJQUMD4GOLO3wVXsXv6LHBRjOS7icqrDm7zdHS0OeUD5usHQPxwwbg0zId%2BgJTKmz6W2S9lSfHGwwv7ukxKraY%2BI3iV00qmq757DeYu0M%2FAEqOa9CyTTCq1TL%2FOqhND1Nm%2BrmqKCp%2FuKqTzz%2BzMEZNUvgA5CTAT7D4M8cg2AL0zlREf8XhnX%2FOrfEwG1mrFN9WyKJavYb13m2pX5%2FellyjGv7PeZ5Q1A2czRjqlt8FdXybAkzCywb%2BCBjrqAfbuvTtMpWQy%2B5VFa2TcyJLuc4bJ2QaGinWaSAAf%2BkOkuVUjcslbodHTDSsHEjI0ezBSvNzcLJe9HUum%2Bqx9YtKrGRqnVFzwhD15D4pD0UN737M8UJk0Zoto%2BQz0ytkhi6ST1SKhu6cJUj0iXyAZd9lpNv7MDhXgUL3dL74ngRqxbyfquO%2BmBGHbfV1%2BP%2B9V3pJqATmXNrNQ6pezbu%2FjdsMXzIO1J%2F8ThEC2JBnq%2FCPCXDNRGJ0l4zqgA35YriFsh%2BmUDvOyB6LPSfJ8DuVpiLU%2BbnyUc1s0WduA6VhlhPTaBlqUznrbMp0fRw%3D%3D&X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Date=20210316T002217Z&X-Amz-SignedHeaders=host&X-Amz-Expires=3599&X-Amz-Credential=ASIA3PSNVSIZYFQQ2SH4%2F20210316%2Fus-west-2%2Fs3%2Faws4_request&X-Amz-Signature=ed2fba6c87fcfe93220e5ce6bc1207ffb9568596c5a73aefd9d83034434c4cf2 [following] --2021-03-16 00:22:17-- https://s3.us-west-2.amazonaws.com/prod-model-registry-ngc-bucket/org/nvidia/recipes/bert_for_tensorflow/versions/1/files.zip?response-content-disposition=attachment%3B%20filename%3D%22files.zip%22&response-content-type=application%2Fzip&X-Amz-Security-Token=IQoJb3JpZ2luX2VjEM%2F%2F%2F%2F%2F%2F%2F%2F%2F%2F%2FwEaCXVzLXdlc3QtMSJIMEYCIQCrGyjQ8LQkZNyTMr6nchH4edRFx%2F25OXI%2FNE2G%2BBF8SQIhAKduF2AEyjRXyw4e5%2BsfnhA81SY%2BN4DOACIaawtQOboyKr0DCPj%2F%2F%2F%2F%2F%2F%2F%2F%2F%2FwEQAxoMNzg5MzYzMTM1MDI3IgxYmy9jrsgCYn3zEQkqkQOLHUCtPKMUjTXYRa2arifO0nTVXmCxHLV%2FCbZmmZOF5J1HGHQgnyZu1y3EPoh8cf5Kn5VfzORhYQlq3KvbKN5%2FXJq0170myZrAOjqrSRUE1DstQl8P0ym3V1S9EWnjYurXuNMa%2FQaqzzZLsVE%2FhuB7N2x0dlI4EZmZ%2B6fnqUlltvMqmc6p1Uh0FQnz6dQ9L1LTTTg2nPx6OAbcbLHjalsPB%2FFBNOSoVRtphUkh4d6iM9uxByZYiYAVr23aJExAU0GAJtjexOZ8bPLzgWcnDRTBzMk0LwKlOtWePK3GFFJQUMD4GOLO3wVXsXv6LHBRjOS7icqrDm7zdHS0OeUD5usHQPxwwbg0zId%2BgJTKmz6W2S9lSfHGwwv7ukxKraY%2BI3iV00qmq757DeYu0M%2FAEqOa9CyTTCq1TL%2FOqhND1Nm%2BrmqKCp%2FuKqTzz%2BzMEZNUvgA5CTAT7D4M8cg2AL0zlREf8XhnX%2FOrfEwG1mrFN9WyKJavYb13m2pX5%2FellyjGv7PeZ5Q1A2czRjqlt8FdXybAkzCywb%2BCBjrqAfbuvTtMpWQy%2B5VFa2TcyJLuc4bJ2QaGinWaSAAf%2BkOkuVUjcslbodHTDSsHEjI0ezBSvNzcLJe9HUum%2Bqx9YtKrGRqnVFzwhD15D4pD0UN737M8UJk0Zoto%2BQz0ytkhi6ST1SKhu6cJUj0iXyAZd9lpNv7MDhXgUL3dL74ngRqxbyfquO%2BmBGHbfV1%2BP%2B9V3pJqATmXNrNQ6pezbu%2FjdsMXzIO1J%2F8ThEC2JBnq%2FCPCXDNRGJ0l4zqgA35YriFsh%2BmUDvOyB6LPSfJ8DuVpiLU%2BbnyUc1s0WduA6VhlhPTaBlqUznrbMp0fRw%3D%3D&X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Date=20210316T002217Z&X-Amz-SignedHeaders=host&X-Amz-Expires=3599&X-Amz-Credential=ASIA3PSNVSIZYFQQ2SH4%2F20210316%2Fus-west-2%2Fs3%2Faws4_request&X-Amz-Signature=ed2fba6c87fcfe93220e5ce6bc1207ffb9568596c5a73aefd9d83034434c4cf2 Resolving s3.us-west-2.amazonaws.com (s3.us-west-2.amazonaws.com)... 52.218.250.24 Connecting to s3.us-west-2.amazonaws.com (s3.us-west-2.amazonaws.com)|52.218.250.24|:443... connected. HTTP request sent, awaiting response... 200 OK Length: 132136 (129K) [application/zip] Saving to: ‘bert_scripts.zip’ bert_scripts.zip 100%[===================>] 129.04K --.-KB/s in 0.1s 2021-03-16 00:22:17 (1.25 MB/s) - ‘bert_scripts.zip’ saved [132136/132136] Archive: bert_scripts.zip inflating: /workspace/bert/.dockerignore inflating: /workspace/bert/CONTRIBUTING.md inflating: /workspace/bert/Dockerfile inflating: /workspace/bert/LICENSE inflating: /workspace/bert/NOTICE inflating: /workspace/bert/README.md inflating: /workspace/bert/__init__.py inflating: /workspace/bert/create_pretraining_data.py inflating: /workspace/bert/data/README.md inflating: /workspace/bert/data/bookcorpus/clean_and_merge_text.py inflating: /workspace/bert/data/bookcorpus/config.sh inflating: /workspace/bert/data/bookcorpus/create_pseudo_test_set.py inflating: /workspace/bert/data/bookcorpus/create_pseudo_test_set.sh inflating: /workspace/bert/data/bookcorpus/preprocessing.sh inflating: /workspace/bert/data/bookcorpus/preprocessing_test_set.sh inflating: /workspace/bert/data/bookcorpus/preprocessing_test_set_xargs_wrapper.sh inflating: /workspace/bert/data/bookcorpus/preprocessing_xargs_wrapper.sh inflating: /workspace/bert/data/bookcorpus/run_preprocessing.sh inflating: /workspace/bert/data/bookcorpus/sentence_segmentation_nltk.py inflating: /workspace/bert/data/bookcorpus/shard_text_input_file.py inflating: /workspace/bert/data/pretrained_models_google/download_models.py inflating: /workspace/bert/data/squad/squad_download.sh inflating: /workspace/bert/data/wikipedia_corpus/config.sh inflating: /workspace/bert/data/wikipedia_corpus/create_pseudo_test_set.py inflating: /workspace/bert/data/wikipedia_corpus/create_pseudo_test_set.sh inflating: /workspace/bert/data/wikipedia_corpus/preprocessing.sh inflating: /workspace/bert/data/wikipedia_corpus/preprocessing_test_set.sh inflating: /workspace/bert/data/wikipedia_corpus/preprocessing_test_set_xargs_wrapper.sh inflating: /workspace/bert/data/wikipedia_corpus/preprocessing_xargs_wrapper.sh inflating: /workspace/bert/data/wikipedia_corpus/remove_tags_and_clean.py inflating: /workspace/bert/data/wikipedia_corpus/run_preprocessing.sh inflating: /workspace/bert/data/wikipedia_corpus/shard_text_input_file.py inflating: /workspace/bert/data/wikipedia_corpus/wiki_sentence_segmentation_nltk.py inflating: /workspace/bert/data/wikipedia_corpus/wiki_sentence_segmentation_spacy.py inflating: /workspace/bert/data/wikipedia_corpus/wiki_sentence_segmentation_spacy_pipe.py inflating: /workspace/bert/extract_features.py inflating: /workspace/bert/gpu_environment.py inflating: /workspace/bert/modeling.py inflating: /workspace/bert/modeling_test.py inflating: /workspace/bert/multilingual.md inflating: /workspace/bert/optimization.py inflating: /workspace/bert/optimization_test.py inflating: /workspace/bert/predicting_movie_reviews_with_bert_on_tf_hub.ipynb inflating: /workspace/bert/requirements.txt inflating: /workspace/bert/run_classifier.py inflating: /workspace/bert/run_classifier_with_tfhub.py inflating: /workspace/bert/run_pretraining.py inflating: /workspace/bert/run_squad.py inflating: /workspace/bert/sample_text.txt inflating: /workspace/bert/scripts/data_download.sh inflating: /workspace/bert/scripts/data_download_helper.sh inflating: /workspace/bert/scripts/docker/build.sh inflating: /workspace/bert/scripts/docker/launch.sh inflating: /workspace/bert/scripts/finetune_inference_benchmark.sh inflating: /workspace/bert/scripts/finetune_train_benchmark.sh inflating: /workspace/bert/scripts/run.sub inflating: /workspace/bert/scripts/run_pretraining.sh inflating: /workspace/bert/scripts/run_squad.sh inflating: /workspace/bert/scripts/run_squad_inference.sh inflating: /workspace/bert/scripts/start_pretraining.sh inflating: /workspace/bert/tokenization.py inflating: /workspace/bert/tokenization_test.py ### BERT Config ```python # Download BERT vocab file !mkdir -p /workspace/bert/config.qa !wget -nc https://s3.amazonaws.com/models.huggingface.co/bert/bert-base-uncased-vocab.txt \ -O /workspace/bert/config.qa/vocab.txt ``` --2021-03-16 00:22:20-- https://s3.amazonaws.com/models.huggingface.co/bert/bert-base-uncased-vocab.txt Resolving s3.amazonaws.com (s3.amazonaws.com)... 52.217.80.246 Connecting to s3.amazonaws.com (s3.amazonaws.com)|52.217.80.246|:443... connected. HTTP request sent, awaiting response... 200 OK Length: 231508 (226K) [text/plain] Saving to: ‘/workspace/bert/config.qa/vocab.txt’ /workspace/bert/con 100%[===================>] 226.08K 364KB/s in 0.6s 2021-03-16 00:22:21 (364 KB/s) - ‘/workspace/bert/config.qa/vocab.txt’ saved [231508/231508] ```python %%writefile /workspace/bert/config.qa/bert_config.json { "attention_probs_dropout_prob": 0.1, "hidden_act": "gelu", "hidden_dropout_prob": 0.1, "hidden_size": 1024, "initializer_range": 0.02, "intermediate_size": 4096, "max_position_embeddings": 512, "num_attention_heads": 16, "num_hidden_layers": 24, "type_vocab_size": 2, "vocab_size": 30522 } ``` Writing /workspace/bert/config.qa/bert_config.json ### Helper Functions ```python # Create dynamic JSON files based on user inputs def write_input_file(context, qinputs, predict_file): # Remove quotes and new lines from text for valid JSON context = context.replace('"', '').replace('\n', '') # Create JSON dict to write json_dict = { "data": [ { "title": "BERT QA", "paragraphs": [ { "context": context, "qas": qinputs } ] } ] } # Write JSON to input file with open(predict_file, 'w') as json_file: import json json.dump(json_dict, json_file, indent=2) # Display Inference Results as HTML Table def display_results(predict_file, output_prediction_file): import json from IPython.display import display, HTML # Here we show only the prediction results, nbest prediction is also available in the output directory results = "" with open(predict_file, 'r') as query_file: queries = json.load(query_file) input_data = queries["data"] with open(output_prediction_file, 'r') as result_file: data = json.load(result_file) for entry in input_data: for paragraph in entry["paragraphs"]: for qa in paragraph["qas"]: results += "{}{}{}".format(qa["id"], qa["question"], data[qa["id"]]) display(HTML("{}

Id	Question	Answer

".format(results))) ``` ## 3. BERT Inference: Question Answering We can run inference on a fine-tuned BERT model for tasks like Question Answering. Here we use a BERT model fine-tuned on a [SQuaD 2.0 Dataset](https://rajpurkar.github.io/SQuAD-explorer/) which contains 100,000+ question-answer pairs on 500+ articles combined with over 50,000 new, unanswerable questions. ### Paragraph and Queries In this example we will ask our BERT model questions related to the following paragraph: **The Apollo Program** _"The Apollo program, also known as Project Apollo, was the third United States human spaceflight program carried out by the National Aeronautics and Space Administration (NASA), which accomplished landing the first humans on the Moon from 1969 to 1972. First conceived during Dwight D. Eisenhower's administration as a three-man spacecraft to follow the one-man Project Mercury which put the first Americans in space, Apollo was later dedicated to President John F. Kennedy's national goal of landing a man on the Moon and returning him safely to the Earth by the end of the 1960s, which he proposed in a May 25, 1961, address to Congress. Project Mercury was followed by the two-man Project Gemini. The first manned flight of Apollo was in 1968. Apollo ran from 1961 to 1972, and was supported by the two-man Gemini program which ran concurrently with it from 1962 to 1966. Gemini missions developed some of the space travel techniques that were necessary for the success of the Apollo missions. Apollo used Saturn family rockets as launch vehicles. Apollo/Saturn vehicles were also used for an Apollo Applications Program, which consisted of Skylab, a space station that supported three manned missions in 1973-74, and the Apollo-Soyuz Test Project, a joint Earth orbit mission with the Soviet Union in 1975."_ --- The paragraph and the questions can be easily customized by changing the code below: --- ```python # Create BERT input file with (1) context and (2) questions to be answered based on that context predict_file = '/workspace/bert/config.qa/input.json' ``` ```python %%writefile $predict_file {"data": [ {"title": "Project Apollo", "paragraphs": [ {"context":"The Apollo program, also known as Project Apollo, was the third United States human spaceflight program carried out by the National Aeronautics and Space Administration (NASA), which accomplished landing the first humans on the Moon from 1969 to 1972. First conceived during Dwight D. Eisenhower's administration as a three-man spacecraft to follow the one-man Project Mercury which put the first Americans in space, Apollo was later dedicated to President John F. Kennedy's national goal of landing a man on the Moon and returning him safely to the Earth by the end of the 1960s, which he proposed in a May 25, 1961, address to Congress. Project Mercury was followed by the two-man Project Gemini. The first manned flight of Apollo was in 1968. Apollo ran from 1961 to 1972, and was supported by the two man Gemini program which ran concurrently with it from 1962 to 1966. Gemini missions developed some of the space travel techniques that were necessary for the success of the Apollo missions. Apollo used Saturn family rockets as launch vehicles. Apollo/Saturn vehicles were also used for an Apollo Applications Program, which consisted of Skylab, a space station that supported three manned missions in 1973-74, and the Apollo-Soyuz Test Project, a joint Earth orbit mission with the Soviet Union in 1975.", "qas": [ { "question": "What project put the first Americans into space?", "id": "Q1" }, { "question": "What program was created to carry out these projects and missions?", "id": "Q2" }, { "question": "What year did the first manned Apollo flight occur?", "id": "Q3" }, { "question": "What President is credited with the notion of putting Americans on the moon?", "id": "Q4" }, { "question": "Who did the U.S. collaborate with on an Earth orbit mission in 1975?", "id": "Q5" }, { "question": "How long did Project Apollo run?", "id": "Q6" }, { "question": "What program helped develop space travel techniques that Project Apollo used?", "id": "Q7" }, {"question": "What space station supported three manned missions in 1973-1974?", "id": "Q8" } ]}]}]} ``` Writing /workspace/bert/config.qa/input.json ## 4. Running Question/Answer Inference To run QA inference we will launch the script run_squad.py with the following parameters: ```python import os # This specifies the model architecture. bert_config_file = '/workspace/bert/config.qa/bert_config.json' # The vocabulary file that the BERT model was trained on. vocab_file = '/workspace/bert/config.qa/vocab.txt' # Initiate checkpoint to the fine-tuned BERT Large model init_checkpoint = os.path.join('/workspace/bert/data/finetuned_large_model_SQUAD2.0/model.ckpt') # Create the output directory where all the results are saved. output_dir = '/workspace/bert/results' output_prediction_file = os.path.join(output_dir,'predictions.json') # Whether to lower case the input - True for uncased models / False for cased models. do_lower_case = True # Total batch size for predictions predict_batch_size = 8 # Whether to run eval on the dev set. do_predict = True # When splitting up a long document into chunks, how much stride to take between chunks. doc_stride = 128 # The maximum total input sequence length after WordPiece tokenization. # Sequences longer than this will be truncated, and sequences shorter than this will be padded. max_seq_length = 384 ``` ### 4a. Run Inference ```python # Ask BERT questions !python /workspace/bert/run_squad.py \ --bert_config_file=$bert_config_file \ --vocab_file=$vocab_file \ --init_checkpoint=$init_checkpoint \ --output_dir=$output_dir \ --do_predict=$do_predict \ --predict_file=$predict_file \ --predict_batch_size=$predict_batch_size \ --doc_stride=$doc_stride \ --max_seq_length=$max_seq_length ``` 2021-03-16 00:22:33.385548: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudart.so.11.0 WARNING:tensorflow:Deprecation warnings have been disabled. Set TF_ENABLE_DEPRECATION_WARNINGS=1 to re-enable them. WARNING:tensorflow:From /workspace/bert/optimization.py:110: The name tf.train.Optimizer is deprecated. Please use tf.compat.v1.train.Optimizer instead. WARNING:tensorflow:From /workspace/bert/run_squad.py:162: The name tf.train.SessionRunHook is deprecated. Please use tf.estimator.SessionRunHook instead. WARNING:tensorflow:From /workspace/bert/run_squad.py:1409: The name tf.app.run is deprecated. Please use tf.compat.v1.app.run instead. WARNING:tensorflow:From /workspace/bert/run_squad.py:1174: The name tf.logging.set_verbosity is deprecated. Please use tf.compat.v1.logging.set_verbosity instead. W0316 00:22:34.840889 140590680020800 module_wrapper.py:137] From /workspace/bert/run_squad.py:1174: The name tf.logging.set_verbosity is deprecated. Please use tf.compat.v1.logging.set_verbosity instead. WARNING:tensorflow:From /workspace/bert/run_squad.py:1174: The name tf.logging.INFO is deprecated. Please use tf.compat.v1.logging.INFO instead. W0316 00:22:34.841060 140590680020800 module_wrapper.py:137] From /workspace/bert/run_squad.py:1174: The name tf.logging.INFO is deprecated. Please use tf.compat.v1.logging.INFO instead. WARNING:tensorflow:From /workspace/bert/modeling.py:94: The name tf.gfile.GFile is deprecated. Please use tf.io.gfile.GFile instead. W0316 00:22:34.841254 140590680020800 module_wrapper.py:137] From /workspace/bert/modeling.py:94: The name tf.gfile.GFile is deprecated. Please use tf.io.gfile.GFile instead. WARNING:tensorflow:From /workspace/bert/run_squad.py:1183: The name tf.gfile.MakeDirs is deprecated. Please use tf.io.gfile.makedirs instead. W0316 00:22:34.842063 140590680020800 module_wrapper.py:137] From /workspace/bert/run_squad.py:1183: The name tf.gfile.MakeDirs is deprecated. Please use tf.io.gfile.makedirs instead. WARNING:tensorflow:From /workspace/bert/run_squad.py:1199: The name tf.ConfigProto is deprecated. Please use tf.compat.v1.ConfigProto instead. W0316 00:22:34.972105 140590680020800 module_wrapper.py:137] From /workspace/bert/run_squad.py:1199: The name tf.ConfigProto is deprecated. Please use tf.compat.v1.ConfigProto instead. WARNING:tensorflow: The TensorFlow contrib module will not be included in TensorFlow 2.0. For more information, please see: * https://github.com/tensorflow/community/blob/master/rfcs/20180907-contrib-sunset.md * https://github.com/tensorflow/addons * https://github.com/tensorflow/io (for I/O related ops) If you depend on functionality not listed there, please file an issue. W0316 00:22:34.972314 140590680020800 lazy_loader.py:50] The TensorFlow contrib module will not be included in TensorFlow 2.0. For more information, please see: * https://github.com/tensorflow/community/blob/master/rfcs/20180907-contrib-sunset.md * https://github.com/tensorflow/addons * https://github.com/tensorflow/io (for I/O related ops) If you depend on functionality not listed there, please file an issue. WARNING:tensorflow:Estimator's model_fn (.model_fn at 0x7fdd090fc3a0>) includes params argument, but params are not passed to Estimator. W0316 00:22:35.720636 140590680020800 estimator.py:1992] Estimator's model_fn (.model_fn at 0x7fdd090fc3a0>) includes params argument, but params are not passed to Estimator. INFO:tensorflow:Using config: {'_model_dir': '/workspace/bert/results', '_tf_random_seed': None, '_save_summary_steps': 100, '_save_checkpoints_steps': 1000, '_save_checkpoints_secs': None, '_session_config': , '_keep_checkpoint_max': 1, '_keep_checkpoint_every_n_hours': 10000, '_log_step_count_steps': None, '_train_distribute': None, '_device_fn': None, '_protocol': None, '_eval_distribute': None, '_experimental_distribute': None, '_experimental_max_worker_delay_secs': None, '_session_creation_timeout_secs': 7200, '_service': None, '_cluster_spec': , '_task_type': 'worker', '_task_id': 0, '_global_id_in_cluster': 0, '_master': '', '_evaluation_master': '', '_is_chief': True, '_num_ps_replicas': 0, '_num_worker_replicas': 1, '_tpu_config': TPUConfig(iterations_per_loop=1000, num_shards=8, num_cores_per_replica=None, per_host_input_for_training=3, tpu_job_name=None, initial_infeed_sleep_secs=None, input_partition_dims=None, eval_training_input_configuration=2, experimental_host_call_every_n_steps=1), '_cluster': None} I0316 00:22:35.721187 140590680020800 estimator.py:212] Using config: {'_model_dir': '/workspace/bert/results', '_tf_random_seed': None, '_save_summary_steps': 100, '_save_checkpoints_steps': 1000, '_save_checkpoints_secs': None, '_session_config': , '_keep_checkpoint_max': 1, '_keep_checkpoint_every_n_hours': 10000, '_log_step_count_steps': None, '_train_distribute': None, '_device_fn': None, '_protocol': None, '_eval_distribute': None, '_experimental_distribute': None, '_experimental_max_worker_delay_secs': None, '_session_creation_timeout_secs': 7200, '_service': None, '_cluster_spec': , '_task_type': 'worker', '_task_id': 0, '_global_id_in_cluster': 0, '_master': '', '_evaluation_master': '', '_is_chief': True, '_num_ps_replicas': 0, '_num_worker_replicas': 1, '_tpu_config': TPUConfig(iterations_per_loop=1000, num_shards=8, num_cores_per_replica=None, per_host_input_for_training=3, tpu_job_name=None, initial_infeed_sleep_secs=None, input_partition_dims=None, eval_training_input_configuration=2, experimental_host_call_every_n_steps=1), '_cluster': None} INFO:tensorflow:_TPUContext: eval_on_tpu True I0316 00:22:35.721480 140590680020800 tpu_context.py:220] _TPUContext: eval_on_tpu True WARNING:tensorflow:eval_on_tpu ignored because use_tpu is False. W0316 00:22:35.721641 140590680020800 tpu_context.py:222] eval_on_tpu ignored because use_tpu is False. WARNING:tensorflow:From /workspace/bert/run_squad.py:266: The name tf.gfile.Open is deprecated. Please use tf.io.gfile.GFile instead. W0316 00:22:35.721812 140590680020800 module_wrapper.py:137] From /workspace/bert/run_squad.py:266: The name tf.gfile.Open is deprecated. Please use tf.io.gfile.GFile instead. WARNING:tensorflow:From /workspace/bert/run_squad.py:1112: The name tf.python_io.TFRecordWriter is deprecated. Please use tf.io.TFRecordWriter instead. W0316 00:22:35.723536 140590680020800 module_wrapper.py:137] From /workspace/bert/run_squad.py:1112: The name tf.python_io.TFRecordWriter is deprecated. Please use tf.io.TFRecordWriter instead. WARNING:tensorflow:From /workspace/bert/run_squad.py:1354: The name tf.logging.info is deprecated. Please use tf.compat.v1.logging.info instead. W0316 00:22:35.794478 140590680020800 module_wrapper.py:137] From /workspace/bert/run_squad.py:1354: The name tf.logging.info is deprecated. Please use tf.compat.v1.logging.info instead. INFO:tensorflow:***** Running predictions ***** I0316 00:22:35.794595 140590680020800 run_squad.py:1354] ***** Running predictions ***** INFO:tensorflow: Num orig examples = 8 I0316 00:22:35.794691 140590680020800 run_squad.py:1355] Num orig examples = 8 INFO:tensorflow: Num split examples = 8 I0316 00:22:35.794868 140590680020800 run_squad.py:1356] Num split examples = 8 INFO:tensorflow: Batch size = 8 I0316 00:22:35.794964 140590680020800 run_squad.py:1357] Batch size = 8 WARNING:tensorflow:From /workspace/bert/run_squad.py:731: The name tf.FixedLenFeature is deprecated. Please use tf.io.FixedLenFeature instead. W0316 00:22:35.795106 140590680020800 module_wrapper.py:137] From /workspace/bert/run_squad.py:731: The name tf.FixedLenFeature is deprecated. Please use tf.io.FixedLenFeature instead. INFO:tensorflow:Could not find trained model in model_dir: /workspace/bert/results, running initialization to predict. I0316 00:22:35.795398 140590680020800 estimator.py:614] Could not find trained model in model_dir: /workspace/bert/results, running initialization to predict. WARNING:tensorflow:Entity .input_fn.. at 0x7fdcff6d63a0> could not be transformed and will be executed as-is. Please report this to the AutoGraph team. When filing the bug, set the verbosity to 10 (on Linux, `export AUTOGRAPH_VERBOSITY=10`) and attach the full output. Cause: 'arguments' object has no attribute 'defaults' W0316 00:22:35.851542 140590680020800 ag_logging.py:146] Entity .input_fn.. at 0x7fdcff6d63a0> could not be transformed and will be executed as-is. Please report this to the AutoGraph team. When filing the bug, set the verbosity to 10 (on Linux, `export AUTOGRAPH_VERBOSITY=10`) and attach the full output. Cause: 'arguments' object has no attribute 'defaults' WARNING:tensorflow:From /workspace/bert/run_squad.py:743: The name tf.parse_single_example is deprecated. Please use tf.io.parse_single_example instead. W0316 00:22:35.851733 140590680020800 module_wrapper.py:137] From /workspace/bert/run_squad.py:743: The name tf.parse_single_example is deprecated. Please use tf.io.parse_single_example instead. INFO:tensorflow:Calling model_fn. I0316 00:22:35.867721 140590680020800 estimator.py:1148] Calling model_fn. INFO:tensorflow:Running infer on CPU I0316 00:22:35.867883 140590680020800 tpu_estimator.py:3124] Running infer on CPU WARNING:tensorflow:From /workspace/bert/modeling.py:175: The name tf.variable_scope is deprecated. Please use tf.compat.v1.variable_scope instead. W0316 00:22:35.870865 140590680020800 module_wrapper.py:137] From /workspace/bert/modeling.py:175: The name tf.variable_scope is deprecated. Please use tf.compat.v1.variable_scope instead. WARNING:tensorflow:From /workspace/bert/modeling.py:413: The name tf.get_variable is deprecated. Please use tf.compat.v1.get_variable instead. W0316 00:22:35.872210 140590680020800 module_wrapper.py:137] From /workspace/bert/modeling.py:413: The name tf.get_variable is deprecated. Please use tf.compat.v1.get_variable instead. WARNING:tensorflow:From /workspace/bert/modeling.py:494: The name tf.assert_less_equal is deprecated. Please use tf.compat.v1.assert_less_equal instead. W0316 00:22:35.897977 140590680020800 module_wrapper.py:137] From /workspace/bert/modeling.py:494: The name tf.assert_less_equal is deprecated. Please use tf.compat.v1.assert_less_equal instead. WARNING:tensorflow:From /workspace/bert/run_squad.py:655: The name tf.trainable_variables is deprecated. Please use tf.compat.v1.trainable_variables instead. W0316 00:22:39.388511 140590680020800 module_wrapper.py:137] From /workspace/bert/run_squad.py:655: The name tf.trainable_variables is deprecated. Please use tf.compat.v1.trainable_variables instead. WARNING:tensorflow:From /workspace/bert/run_squad.py:670: The name tf.train.init_from_checkpoint is deprecated. Please use tf.compat.v1.train.init_from_checkpoint instead. W0316 00:22:39.394683 140590680020800 module_wrapper.py:137] From /workspace/bert/run_squad.py:670: The name tf.train.init_from_checkpoint is deprecated. Please use tf.compat.v1.train.init_from_checkpoint instead. INFO:tensorflow:Done calling model_fn. I0316 00:22:40.470538 140590680020800 estimator.py:1150] Done calling model_fn. INFO:tensorflow:Graph was finalized. I0316 00:22:41.255115 140590680020800 monitored_session.py:240] Graph was finalized. 2021-03-16 00:22:41.283573: I tensorflow/core/platform/profile_utils/cpu_utils.cc:94] CPU Frequency: 2198615000 Hz 2021-03-16 00:22:41.286205: I tensorflow/compiler/xla/service/service.cc:168] XLA service 0x54540d0 initialized for platform Host (this does not guarantee that XLA will be used). Devices: 2021-03-16 00:22:41.286256: I tensorflow/compiler/xla/service/service.cc:176] StreamExecutor device (0): Host, Default Version 2021-03-16 00:22:41.290408: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcuda.so.1 2021-03-16 00:22:41.533042: I tensorflow/compiler/xla/service/service.cc:168] XLA service 0x54339f0 initialized for platform CUDA (this does not guarantee that XLA will be used). Devices: 2021-03-16 00:22:41.533093: I tensorflow/compiler/xla/service/service.cc:176] StreamExecutor device (0): Tesla V100-DGXS-32GB, Compute Capability 7.0 2021-03-16 00:22:41.537140: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1665] Found device 0 with properties: name: Tesla V100-DGXS-32GB major: 7 minor: 0 memoryClockRate(GHz): 1.53 pciBusID: 0000:08:00.0 2021-03-16 00:22:41.537196: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudart.so.11.0 2021-03-16 00:22:41.554704: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcublas.so.11 2021-03-16 00:22:41.562981: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcufft.so.10 2021-03-16 00:22:41.565560: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcurand.so.10 2021-03-16 00:22:41.581478: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcusolver.so.11 2021-03-16 00:22:41.585296: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcusparse.so.11 2021-03-16 00:22:41.585491: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudnn.so.8 2021-03-16 00:22:41.590070: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1793] Adding visible gpu devices: 0 2021-03-16 00:22:41.590119: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudart.so.11.0 2021-03-16 00:22:42.007522: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1206] Device interconnect StreamExecutor with strength 1 edge matrix: 2021-03-16 00:22:42.007563: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1212] 0 2021-03-16 00:22:42.007576: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1225] 0: N 2021-03-16 00:22:42.011538: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1351] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 31026 MB memory) -> physical GPU (device: 0, name: Tesla V100-DGXS-32GB, pci bus id: 0000:08:00.0, compute capability: 7.0) INFO:tensorflow:Running local_init_op. I0316 00:22:44.737054 140590680020800 session_manager.py:500] Running local_init_op. INFO:tensorflow:Done running local_init_op. I0316 00:22:44.841894 140590680020800 session_manager.py:502] Done running local_init_op. 2021-03-16 00:22:46.448377: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcublas.so.11 INFO:tensorflow:Processing example: 0 I0316 00:22:47.535837 140590680020800 run_squad.py:1373] Processing example: 0 INFO:tensorflow:prediction_loop marked as finished I0316 00:22:47.610052 140590680020800 error_handling.py:101] prediction_loop marked as finished INFO:tensorflow:prediction_loop marked as finished I0316 00:22:47.610238 140590680020800 error_handling.py:101] prediction_loop marked as finished INFO:tensorflow:----------------------------- I0316 00:22:47.610357 140590680020800 run_squad.py:1388] ----------------------------- INFO:tensorflow:0 Total Inference Time = 11.82 Inference Time W/O start up overhead = 2.01 Sentences processed = 8 I0316 00:22:47.610444 140590680020800 run_squad.py:1389] 0 Total Inference Time = 11.82 Inference Time W/O start up overhead = 2.01 Sentences processed = 8 INFO:tensorflow:0 Inference Performance = 3.9767 sentences/sec I0316 00:22:47.610535 140590680020800 run_squad.py:1392] 0 Inference Performance = 3.9767 sentences/sec INFO:tensorflow:----------------------------- I0316 00:22:47.610617 140590680020800 run_squad.py:1393] ----------------------------- INFO:tensorflow:Writing predictions to: /workspace/bert/results/predictions.json I0316 00:22:47.610741 140590680020800 run_squad.py:792] Writing predictions to: /workspace/bert/results/predictions.json INFO:tensorflow:Writing nbest to: /workspace/bert/results/nbest_predictions.json I0316 00:22:47.610820 140590680020800 run_squad.py:793] Writing nbest to: /workspace/bert/results/nbest_predictions.json ### 4b. Display Results: ```python display_results(predict_file, output_prediction_file) ```

Id	Question	Answer
Q1	What project put the first Americans into space?	Project Mercury
Q2	What program was created to carry out these projects and missions?	The Apollo program
Q3	What year did the first manned Apollo flight occur?	1968
Q4	What President is credited with the notion of putting Americans on the moon?	John F. Kennedy
Q5	Who did the U.S. collaborate with on an Earth orbit mission in 1975?	Soviet Union
Q6	How long did Project Apollo run?	1961 to 1972
Q7	What program helped develop space travel techniques that Project Apollo used?	Gemini missions
Q8	What space station supported three manned missions in 1973-1974?	Skylab

Click to reveal expected answers to the questions above

| Id | Question | Answer | |----|----------|--------| | Q1 | What project put the first Americans into space? | Project Mercury | | Q2 | What program was created to carry out these projects and missions? | The Apollo program | | Q3 | What year did the first manned Apollo flight occur? | 1968 | | Q4 | What President is credited with the notion of putting Americans on the moon? | John F. Kennedy | | Q5 | Who did the U.S. collaborate with on an Earth orbit mission in 1975? | Soviet Union | | Q6 | How long did Project Apollo run? | 1961 to 1972 | | Q7 | What program helped develop space travel techniques that Project Apollo used? | Gemini missions | | Q8 | What space station supported three manned missions in 1973-1974? | Skylab |

## 5. Custom Inputs Now that you are familiar with running QA Inference on BERT, you may want to try your own paragraphs and queries. 1. Copy and paste your context from Wikipedia, news articles, etc. when prompted below 2. Enter questions based on the context when prompted below. 3. Run the inference script 4. Display the inference results ```python predict_file = '/workspace/bert/config.qa/custom_input.json' num_questions = 3 # You can configure this number ``` ```python # Create your own context to ask questions about. context = input("Paste your context here: ") ``` Paste your context here: Alan Mathison Turing OBE FRS (/ˈtjʊərɪŋ/; 23 June 1912 – 7 June 1954) was an English[6] mathematician, computer scientist, logician, cryptanalyst, philosopher and theoretical biologist.[7] Turing was highly influential in the development of theoretical computer science, providing a formalisation of the concepts of algorithm and computation with the Turing machine, which can be considered a model of a general-purpose computer.[8][9][10] Turing is widely considered to be the father of theoretical computer science and artificial intelligence.[11] Despite these accomplishments, he was not fully recognised in his home country during his lifetime, due to his homosexuality, and because much of his work was covered by the Official Secrets Act. During the Second World War, Turing worked for the Government Code and Cypher School (GC&CS) at Bletchley Park, Britain's codebreaking centre that produced Ultra intelligence. For a time he led Hut 8, the section that was responsible for German naval cryptanalysis. Here, he devised a number of techniques for speeding the breaking of German ciphers, including improvements to the pre-war Polish bombe method, an electromechanical machine that could find settings for the Enigma machine. Turing played a pivotal role in cracking intercepted coded messages that enabled the Allies to defeat the Nazis in many crucial engagements, including the Battle of the Atlantic, and in so doing helped win the war.[12][13] Due to the problems of counterfactual history, it's hard to estimate what effect Ultra intelligence had on the war,[14] but at the upper end it has been estimated that this work shortened the war in Europe by more than two years and saved over 14 million lives.[12] After the war, Turing worked at the National Physical Laboratory, where he designed the Automatic Computing Engine, which was one of the first designs for a stored-program computer. In 1948, Turing joined Max Newman's Computing Machine Laboratory at the Victoria University of Manchester, where he helped develop the Manchester computers[15] and became interested in mathematical biology. He wrote a paper on the chemical basis of morphogenesis[1] and predicted oscillating chemical reactions such as the Belousov–Zhabotinsky reaction, first observed in the 1960s. Turing was prosecuted in 1952 for homosexual acts; the Labouchere Amendment of 1885 had mandated that "gross indecency" was a criminal offence in the UK. He accepted chemical castration treatment, with DES, as an alternative to prison. Turing died in 1954, 16 days before his 42nd birthday, from cyanide poisoning. An inquest determined his death as a suicide, but it has been noted that the known evidence is also consistent with accidental poisoning. In 2009, following an Internet campaign, British Prime Minister Gordon Brown made an official public apology on behalf of the British government for "the appalling way he was treated". Queen Elizabeth II granted Turing a posthumous pardon in 2013. The Alan Turing law is now an informal term for a 2017 law in the United Kingdom that retroactively pardoned men cautioned or convicted under historical legislation that outlawed homosexual acts.[16] On 15 July 2019 the Bank of England announced that Turing would be depicted on the United Kingdom's new £50 note. ```python # Get questions from user input questions = [input("Question {}/{}: ".format(i+1, num_questions)) for i in range(num_questions)] # Format questions and write to JSON input file qinputs = [{ "question":q, "id":"Q{}".format(i+1)} for i,q in enumerate(questions)] write_input_file(context, qinputs, predict_file) ``` Question 1/3: Who is Alan Turing? Question 2/3: When was Alan Turing born? Question 3/3: Which laboratory Turing joined in 1948? ```python # Ask BERT questions !python /workspace/bert/run_squad.py \ --bert_config_file=$bert_config_file \ --vocab_file=$vocab_file \ --init_checkpoint=$init_checkpoint \ --output_dir=$output_dir \ --do_predict=$do_predict \ --predict_file=$predict_file \ --predict_batch_size=$predict_batch_size \ --doc_stride=$doc_stride \ --max_seq_length=$max_seq_length ``` 2021-03-16 00:25:20.406731: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudart.so.11.0 WARNING:tensorflow:Deprecation warnings have been disabled. Set TF_ENABLE_DEPRECATION_WARNINGS=1 to re-enable them. WARNING:tensorflow:From /workspace/bert/optimization.py:110: The name tf.train.Optimizer is deprecated. Please use tf.compat.v1.train.Optimizer instead. WARNING:tensorflow:From /workspace/bert/run_squad.py:162: The name tf.train.SessionRunHook is deprecated. Please use tf.estimator.SessionRunHook instead. WARNING:tensorflow:From /workspace/bert/run_squad.py:1409: The name tf.app.run is deprecated. Please use tf.compat.v1.app.run instead. WARNING:tensorflow:From /workspace/bert/run_squad.py:1174: The name tf.logging.set_verbosity is deprecated. Please use tf.compat.v1.logging.set_verbosity instead. W0316 00:25:21.868111 140608677435200 module_wrapper.py:137] From /workspace/bert/run_squad.py:1174: The name tf.logging.set_verbosity is deprecated. Please use tf.compat.v1.logging.set_verbosity instead. WARNING:tensorflow:From /workspace/bert/run_squad.py:1174: The name tf.logging.INFO is deprecated. Please use tf.compat.v1.logging.INFO instead. W0316 00:25:21.868280 140608677435200 module_wrapper.py:137] From /workspace/bert/run_squad.py:1174: The name tf.logging.INFO is deprecated. Please use tf.compat.v1.logging.INFO instead. WARNING:tensorflow:From /workspace/bert/modeling.py:94: The name tf.gfile.GFile is deprecated. Please use tf.io.gfile.GFile instead. W0316 00:25:21.868470 140608677435200 module_wrapper.py:137] From /workspace/bert/modeling.py:94: The name tf.gfile.GFile is deprecated. Please use tf.io.gfile.GFile instead. WARNING:tensorflow:From /workspace/bert/run_squad.py:1183: The name tf.gfile.MakeDirs is deprecated. Please use tf.io.gfile.makedirs instead. W0316 00:25:21.869253 140608677435200 module_wrapper.py:137] From /workspace/bert/run_squad.py:1183: The name tf.gfile.MakeDirs is deprecated. Please use tf.io.gfile.makedirs instead. WARNING:tensorflow:From /workspace/bert/run_squad.py:1199: The name tf.ConfigProto is deprecated. Please use tf.compat.v1.ConfigProto instead. W0316 00:25:21.998484 140608677435200 module_wrapper.py:137] From /workspace/bert/run_squad.py:1199: The name tf.ConfigProto is deprecated. Please use tf.compat.v1.ConfigProto instead. WARNING:tensorflow: The TensorFlow contrib module will not be included in TensorFlow 2.0. For more information, please see: * https://github.com/tensorflow/community/blob/master/rfcs/20180907-contrib-sunset.md * https://github.com/tensorflow/addons * https://github.com/tensorflow/io (for I/O related ops) If you depend on functionality not listed there, please file an issue. W0316 00:25:21.998684 140608677435200 lazy_loader.py:50] The TensorFlow contrib module will not be included in TensorFlow 2.0. For more information, please see: * https://github.com/tensorflow/community/blob/master/rfcs/20180907-contrib-sunset.md * https://github.com/tensorflow/addons * https://github.com/tensorflow/io (for I/O related ops) If you depend on functionality not listed there, please file an issue. WARNING:tensorflow:Estimator's model_fn (.model_fn at 0x7fe139ce63a0>) includes params argument, but params are not passed to Estimator. W0316 00:25:22.744968 140608677435200 estimator.py:1992] Estimator's model_fn (.model_fn at 0x7fe139ce63a0>) includes params argument, but params are not passed to Estimator. INFO:tensorflow:Using config: {'_model_dir': '/workspace/bert/results', '_tf_random_seed': None, '_save_summary_steps': 100, '_save_checkpoints_steps': 1000, '_save_checkpoints_secs': None, '_session_config': , '_keep_checkpoint_max': 1, '_keep_checkpoint_every_n_hours': 10000, '_log_step_count_steps': None, '_train_distribute': None, '_device_fn': None, '_protocol': None, '_eval_distribute': None, '_experimental_distribute': None, '_experimental_max_worker_delay_secs': None, '_session_creation_timeout_secs': 7200, '_service': None, '_cluster_spec': , '_task_type': 'worker', '_task_id': 0, '_global_id_in_cluster': 0, '_master': '', '_evaluation_master': '', '_is_chief': True, '_num_ps_replicas': 0, '_num_worker_replicas': 1, '_tpu_config': TPUConfig(iterations_per_loop=1000, num_shards=8, num_cores_per_replica=None, per_host_input_for_training=3, tpu_job_name=None, initial_infeed_sleep_secs=None, input_partition_dims=None, eval_training_input_configuration=2, experimental_host_call_every_n_steps=1), '_cluster': None} I0316 00:25:22.745511 140608677435200 estimator.py:212] Using config: {'_model_dir': '/workspace/bert/results', '_tf_random_seed': None, '_save_summary_steps': 100, '_save_checkpoints_steps': 1000, '_save_checkpoints_secs': None, '_session_config': , '_keep_checkpoint_max': 1, '_keep_checkpoint_every_n_hours': 10000, '_log_step_count_steps': None, '_train_distribute': None, '_device_fn': None, '_protocol': None, '_eval_distribute': None, '_experimental_distribute': None, '_experimental_max_worker_delay_secs': None, '_session_creation_timeout_secs': 7200, '_service': None, '_cluster_spec': , '_task_type': 'worker', '_task_id': 0, '_global_id_in_cluster': 0, '_master': '', '_evaluation_master': '', '_is_chief': True, '_num_ps_replicas': 0, '_num_worker_replicas': 1, '_tpu_config': TPUConfig(iterations_per_loop=1000, num_shards=8, num_cores_per_replica=None, per_host_input_for_training=3, tpu_job_name=None, initial_infeed_sleep_secs=None, input_partition_dims=None, eval_training_input_configuration=2, experimental_host_call_every_n_steps=1), '_cluster': None} INFO:tensorflow:_TPUContext: eval_on_tpu True I0316 00:25:22.745804 140608677435200 tpu_context.py:220] _TPUContext: eval_on_tpu True WARNING:tensorflow:eval_on_tpu ignored because use_tpu is False. W0316 00:25:22.745965 140608677435200 tpu_context.py:222] eval_on_tpu ignored because use_tpu is False. WARNING:tensorflow:From /workspace/bert/run_squad.py:266: The name tf.gfile.Open is deprecated. Please use tf.io.gfile.GFile instead. W0316 00:25:22.746137 140608677435200 module_wrapper.py:137] From /workspace/bert/run_squad.py:266: The name tf.gfile.Open is deprecated. Please use tf.io.gfile.GFile instead. WARNING:tensorflow:From /workspace/bert/run_squad.py:1112: The name tf.python_io.TFRecordWriter is deprecated. Please use tf.io.TFRecordWriter instead. W0316 00:25:22.749643 140608677435200 module_wrapper.py:137] From /workspace/bert/run_squad.py:1112: The name tf.python_io.TFRecordWriter is deprecated. Please use tf.io.TFRecordWriter instead. WARNING:tensorflow:From /workspace/bert/run_squad.py:1354: The name tf.logging.info is deprecated. Please use tf.compat.v1.logging.info instead. W0316 00:25:22.830683 140608677435200 module_wrapper.py:137] From /workspace/bert/run_squad.py:1354: The name tf.logging.info is deprecated. Please use tf.compat.v1.logging.info instead. INFO:tensorflow:***** Running predictions ***** I0316 00:25:22.830802 140608677435200 run_squad.py:1354] ***** Running predictions ***** INFO:tensorflow: Num orig examples = 3 I0316 00:25:22.830897 140608677435200 run_squad.py:1355] Num orig examples = 3 INFO:tensorflow: Num split examples = 12 I0316 00:25:22.831071 140608677435200 run_squad.py:1356] Num split examples = 12 INFO:tensorflow: Batch size = 8 I0316 00:25:22.831164 140608677435200 run_squad.py:1357] Batch size = 8 WARNING:tensorflow:From /workspace/bert/run_squad.py:731: The name tf.FixedLenFeature is deprecated. Please use tf.io.FixedLenFeature instead. W0316 00:25:22.831306 140608677435200 module_wrapper.py:137] From /workspace/bert/run_squad.py:731: The name tf.FixedLenFeature is deprecated. Please use tf.io.FixedLenFeature instead. INFO:tensorflow:Could not find trained model in model_dir: /workspace/bert/results, running initialization to predict. I0316 00:25:22.831603 140608677435200 estimator.py:614] Could not find trained model in model_dir: /workspace/bert/results, running initialization to predict. WARNING:tensorflow:Entity .input_fn.. at 0x7fe1302d1a60> could not be transformed and will be executed as-is. Please report this to the AutoGraph team. When filing the bug, set the verbosity to 10 (on Linux, `export AUTOGRAPH_VERBOSITY=10`) and attach the full output. Cause: 'arguments' object has no attribute 'defaults' W0316 00:25:22.888306 140608677435200 ag_logging.py:146] Entity .input_fn.. at 0x7fe1302d1a60> could not be transformed and will be executed as-is. Please report this to the AutoGraph team. When filing the bug, set the verbosity to 10 (on Linux, `export AUTOGRAPH_VERBOSITY=10`) and attach the full output. Cause: 'arguments' object has no attribute 'defaults' WARNING:tensorflow:From /workspace/bert/run_squad.py:743: The name tf.parse_single_example is deprecated. Please use tf.io.parse_single_example instead. W0316 00:25:22.888502 140608677435200 module_wrapper.py:137] From /workspace/bert/run_squad.py:743: The name tf.parse_single_example is deprecated. Please use tf.io.parse_single_example instead. INFO:tensorflow:Calling model_fn. I0316 00:25:22.904676 140608677435200 estimator.py:1148] Calling model_fn. INFO:tensorflow:Running infer on CPU I0316 00:25:22.904836 140608677435200 tpu_estimator.py:3124] Running infer on CPU WARNING:tensorflow:From /workspace/bert/modeling.py:175: The name tf.variable_scope is deprecated. Please use tf.compat.v1.variable_scope instead. W0316 00:25:22.907857 140608677435200 module_wrapper.py:137] From /workspace/bert/modeling.py:175: The name tf.variable_scope is deprecated. Please use tf.compat.v1.variable_scope instead. WARNING:tensorflow:From /workspace/bert/modeling.py:413: The name tf.get_variable is deprecated. Please use tf.compat.v1.get_variable instead. W0316 00:25:22.909193 140608677435200 module_wrapper.py:137] From /workspace/bert/modeling.py:413: The name tf.get_variable is deprecated. Please use tf.compat.v1.get_variable instead. WARNING:tensorflow:From /workspace/bert/modeling.py:494: The name tf.assert_less_equal is deprecated. Please use tf.compat.v1.assert_less_equal instead. W0316 00:25:22.935186 140608677435200 module_wrapper.py:137] From /workspace/bert/modeling.py:494: The name tf.assert_less_equal is deprecated. Please use tf.compat.v1.assert_less_equal instead. WARNING:tensorflow:From /workspace/bert/run_squad.py:655: The name tf.trainable_variables is deprecated. Please use tf.compat.v1.trainable_variables instead. W0316 00:25:26.434307 140608677435200 module_wrapper.py:137] From /workspace/bert/run_squad.py:655: The name tf.trainable_variables is deprecated. Please use tf.compat.v1.trainable_variables instead. WARNING:tensorflow:From /workspace/bert/run_squad.py:670: The name tf.train.init_from_checkpoint is deprecated. Please use tf.compat.v1.train.init_from_checkpoint instead. W0316 00:25:26.440835 140608677435200 module_wrapper.py:137] From /workspace/bert/run_squad.py:670: The name tf.train.init_from_checkpoint is deprecated. Please use tf.compat.v1.train.init_from_checkpoint instead. INFO:tensorflow:Done calling model_fn. I0316 00:25:27.519404 140608677435200 estimator.py:1150] Done calling model_fn. INFO:tensorflow:Graph was finalized. I0316 00:25:28.309463 140608677435200 monitored_session.py:240] Graph was finalized. 2021-03-16 00:25:28.339586: I tensorflow/core/platform/profile_utils/cpu_utils.cc:94] CPU Frequency: 2198615000 Hz 2021-03-16 00:25:28.341293: I tensorflow/compiler/xla/service/service.cc:168] XLA service 0x5978d70 initialized for platform Host (this does not guarantee that XLA will be used). Devices: 2021-03-16 00:25:28.341321: I tensorflow/compiler/xla/service/service.cc:176] StreamExecutor device (0): Host, Default Version 2021-03-16 00:25:28.344109: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcuda.so.1 2021-03-16 00:25:28.580121: I tensorflow/compiler/xla/service/service.cc:168] XLA service 0x8b42f70 initialized for platform CUDA (this does not guarantee that XLA will be used). Devices: 2021-03-16 00:25:28.580169: I tensorflow/compiler/xla/service/service.cc:176] StreamExecutor device (0): Tesla V100-DGXS-32GB, Compute Capability 7.0 2021-03-16 00:25:28.583618: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1665] Found device 0 with properties: name: Tesla V100-DGXS-32GB major: 7 minor: 0 memoryClockRate(GHz): 1.53 pciBusID: 0000:08:00.0 2021-03-16 00:25:28.583675: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudart.so.11.0 2021-03-16 00:25:28.589475: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcublas.so.11 2021-03-16 00:25:28.591388: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcufft.so.10 2021-03-16 00:25:28.591844: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcurand.so.10 2021-03-16 00:25:28.596971: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcusolver.so.11 2021-03-16 00:25:28.597815: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcusparse.so.11 2021-03-16 00:25:28.597983: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudnn.so.8 2021-03-16 00:25:28.601731: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1793] Adding visible gpu devices: 0 2021-03-16 00:25:28.601770: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudart.so.11.0 2021-03-16 00:25:28.994633: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1206] Device interconnect StreamExecutor with strength 1 edge matrix: 2021-03-16 00:25:28.994679: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1212] 0 2021-03-16 00:25:28.994692: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1225] 0: N 2021-03-16 00:25:28.998624: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1351] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 31026 MB memory) -> physical GPU (device: 0, name: Tesla V100-DGXS-32GB, pci bus id: 0000:08:00.0, compute capability: 7.0) INFO:tensorflow:Running local_init_op. I0316 00:25:31.745624 140608677435200 session_manager.py:500] Running local_init_op. INFO:tensorflow:Done running local_init_op. I0316 00:25:31.858720 140608677435200 session_manager.py:502] Done running local_init_op. 2021-03-16 00:25:33.689415: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcublas.so.11 INFO:tensorflow:Processing example: 0 I0316 00:25:34.384404 140608677435200 run_squad.py:1373] Processing example: 0 INFO:tensorflow:prediction_loop marked as finished I0316 00:25:34.586190 140608677435200 error_handling.py:101] prediction_loop marked as finished INFO:tensorflow:prediction_loop marked as finished I0316 00:25:34.586374 140608677435200 error_handling.py:101] prediction_loop marked as finished INFO:tensorflow:----------------------------- I0316 00:25:34.586491 140608677435200 run_squad.py:1388] ----------------------------- INFO:tensorflow:0 Total Inference Time = 11.76 Inference Time W/O start up overhead = 1.89 Sentences processed = 16 I0316 00:25:34.586577 140608677435200 run_squad.py:1389] 0 Total Inference Time = 11.76 Inference Time W/O start up overhead = 1.89 Sentences processed = 16 INFO:tensorflow:0 Inference Performance = 8.4703 sentences/sec I0316 00:25:34.586667 140608677435200 run_squad.py:1392] 0 Inference Performance = 8.4703 sentences/sec INFO:tensorflow:----------------------------- I0316 00:25:34.586749 140608677435200 run_squad.py:1393] ----------------------------- INFO:tensorflow:Writing predictions to: /workspace/bert/results/predictions.json I0316 00:25:34.586873 140608677435200 run_squad.py:792] Writing predictions to: /workspace/bert/results/predictions.json INFO:tensorflow:Writing nbest to: /workspace/bert/results/nbest_predictions.json I0316 00:25:34.586952 140608677435200 run_squad.py:793] Writing nbest to: /workspace/bert/results/nbest_predictions.json ```python display_results(predict_file, output_prediction_file) ```

Id	Question	Answer
Q1	Who is Alan Turing?	father of theoretical computer science and artificial intelligence
Q2	When was Alan Turing born?	23 June 1912
Q3	Which laboratory Turing joined in 1948?	Max Newman's Computing Machine Laboratory at the Victoria University of Manchester

```python ```