PythonでベクトルインデックスとOpenAIを使って質問応答システムを構築する方法

pip install momento openai

import openai
import requests
from momento import CredentialProvider, PreviewVectorIndexClient, VectorIndexConfigurations
from momento.config import VectorIndexConfiguration
from momento.requests.vector_index import Item
from momento.responses.vector_index import Search, UpsertItemBatch
import os

# Setting up the API keys and clients
openai.api_key = os.environ['OPENAI_API_KEY']
VECTOR_INDEX_CONFIGURATION: VectorIndexConfiguration = VectorIndexConfigurations.Default.latest()
VECTOR_AUTH_PROVIDER = CredentialProvider.from_environment_variable('MOMENTO_API_KEY')
mvi_client = PreviewVectorIndexClient(VECTOR_INDEX_CONFIGURATION, VECTOR_AUTH_PROVIDER)
index_name = 'mvi-openai-demo'

def get_wikipedia_extract(url: str) -> str:
    response = requests.get(url)
    data = response.json()
    pages = data['query']['pages']
    extract = next(iter(pages.values()))['extract']
    return extract

url = "https://en.wikipedia.org/w/api.php?action=query&format=json&titles=Carrot&prop=extracts&explaintext";

extract_text = get_wikipedia_extract(url)
print('Total characters in carrot Wikipedia page: ' + str(len(extract_text)))
print('Sample text in carrot Wikipedia page:\n\n ' + extract_text[0:500])

Total characters in carrot Wikipedia page: 21534

The carrot (Daucus carota subsp. sativus) is a root vegetable, typically orange in color, though heirloom variants including purple, black, red, white, and yellow cultivars exist, all of which are domesticated forms of the wild carrot, Daucus carota, native to Europe and Southwestern Asia. The plant probably originated in Persia and was originally cultivated for its leaves and seeds. The most commonly eaten part of the plant is the taproot, although the stems and leaves are also eaten.

def split_text_into_chunks(text: str, chunk_size: int = 600) -> list[str]:
    return [text[i:i + chunk_size] for i in range(0, len(text), chunk_size)]

chunks = split_text_into_chunks(extract_text)

print('Total number of chunks created: ' + str(len(chunks)))
print('Total characters in each chunk: ' + str(len(chunks[0])))

Total number of chunks created: 36
Total characters in each chunk: 600

def generate_embeddings(chunks: list[str]):
    response = openai.embeddings.create(input=chunks, model="text-embedding-ada-002")
    return response.data

embeddings_response = generate_embeddings(chunks)
print('Length of each embedding: ' + str(len(embeddings_response[0].embedding)))
print('Sample embedding: ' + str(embeddings_response[0].embedding[0:10]))

Length of each embedding: 1536

Sample embedding: 0.008307404,-0.03437371,0.00043777542,-0.01768263,-0.010926112,-0.0056728064,-0.0025742147,-0.023453956,-0.021114917,-0.020148791

def upsert_to_mvi(embeddings: list, chunks: list[str]):

    metadatas = [{"text": chunk} for chunk in chunks]

    ids = [f"chunk{i + 1}" for i, _ in enumerate(embeddings)]

    items = [Item(id=id, vector=embedding.embedding, metadata=metadata) for id, embedding, metadata in zip(ids, embeddings, metadatas)]

    response = mvi_client.upsert_item_batch(index_name, items)

    if (isinstance(response, UpsertItemBatch.Success)):
        print("\n\nUpsert successful. Items have been stored.")
    elif isinstance(response, UpsertItemBatch.Error):
        print(response.message)
        raise response.inner_exception

upsert_to_mvi(embeddings_response, chunks)

Upsert successful. Items have been stored.

def search_query(query_text: str) -> list[str]:
    query_vector = openai.embeddings.create(input=query_text, model="text-embedding-ada-002").data[0].embedding
    search_response = mvi_client.search(index_name, query_vector=query_vector, top_k=2, metadata_fields=["text"])
    if isinstance(search_response, Search.Success):
        return [hit.metadata['text'] for hit in search_response.hits]
    elif isinstance(search_response, Search.Error):
        print(f"Error while searching on index {index_name}: {search_response.message}")
        return []

query = "What is a carrot?"
texts = search_query(query, index_name)
if texts:
    print("\n=========================================\n”)
    print("Embedding search results:\n\n" + "\n".join(texts))
    print("\n=========================================\n")

The carrot (Daucus carota subsp. sativus) is a root vegetable, typically orange in color, though heirloom variants including purple, black, red, white, and yellow cultivars exist, all of which are domesticated forms of the wild carrot, Daucus carota, native to Europe and Southwestern Asia. The plant probably originated in Persia and was originally cultivated for its leaves and seeds. The most commonly eaten part of the plant is the taproot, although the stems and leaves are also eaten. The domestic carrot has been selectively bred for its enlarged, more palatable, less woody-textured taproot.

The carrot is a biennial plant in the umbellifer family, Apiaceae. At birth, it grows a rosette of leaves while building up the enlarged taproot. Fast-growing cultivars mature within about three months (90 days) of sowing the seed, while slower-maturing cultivars need a month longer (120 days). The roots contain high quantities of alpha- and beta-carotene, lycopene, anthocyanins, lutein, and are a good source of vitamin A, vitamin K, and vitamin B6. Black carrots are one of the richest sources of anthocyanins (250-300 mg/100 g fresh root weight), and hence possesses high antioxidant ability

def search_with_chat_completion(texts: list[str], query_text: str):
    text = "\n".join(texts)
    prompt = ("Given the following extracted parts about carrot, answer questions pertaining to"
              " carrot only from the provided text. If you don't know the answer, just say that "
              "you don't know. Don't try to make up an answer. Do not answer anything outside of the context given. "
              "Your job is to only answer about carrots, and only from the text below. If you don't know the answer, just "
              "say that you don't know. Here's the text:\n\n----------------\n\n")
    chat_response = openai.chat.completions.create(model="gpt-3.5-turbo", messages=[
        {"role": "system", "content": prompt + text},
        {"role": "user", "content": query_text}
    ])
    return chat_response.choices[0].message

chat_completion_resp = search_with_chat_completion(texts, query)
print("\n=========================================\n")
print("Chat completion search results:\n\n" + chat_completion_resp.content)
print("\n=========================================\n")

A carrot is a root vegetable that is typically orange in color, although there are also other colored variants such as purple, black, red, white, and yellow. 

query = "how fast do fast-growing cultivators mature in carrots?"
texts = search_query(query)
if texts:
    print("\n=========================================\n")
    print("Embedding search results:\n\n" + texts[0])
    print("\n=========================================\n")

    chat_completion_resp = search_with_chat_completion(texts, query)
    print("\n=========================================\n")
    print("Chat completion search results:\n\n" + chat_completion_resp.content)
    print("\n=========================================\n")

=========================================
Embedding search results:

The carrot is a biennial plant in the umbellifer family, Apiaceae. At birth, it grows a rosette of leaves while building up the enlarged taproot. Fast-growing cultivars mature within about three months (90 days) of sowing the seed, while slower-maturing cultivars need a month longer (120 days). The roots contain high quantities of alpha- and beta-carotene, lycopene, anthocyanins, lutein, and are a good source of vitamin A, vitamin K, and vitamin B6. Black carrots are one of the richest sources of anthocyanins (250-300 mg/100 g fresh root weight), and hence possesses high antioxidant ability.

=========================================


=========================================
Chat completion search results:

Fast-growing cultivars of carrots mature within about three months (90 days) of sowing the seed.

=========================================