TLDR;

This project turns the entire Bible into a vector space, precomputes semantic cross-references between verses, and lets users explore them visually as a galaxy.

Features to Add Later

• Support for multiple languages and translation versions
• Better dimensionality reduction techniques to represent the verse embeddings in 3D
• Faster query search using ANN algorithms like IVF-PQ or HNSW
• Mobile Friendly website

Journal Entry

This project began as I was studying the bible. Often, when I was analyzing a verse, I would need to reference other verses to further reflect on its meaning. This is usually done through cross references, but I thought that sentence embedding could be used to achieve a similar result.

First, I obtained a dataset of the NKJV bible and cleaned the csv file to have the necessary information. Then, using sentence-transformer library, I converted all the bible verses as an embedding of 384 dimensions. All data for each bible verse, including the embedding was stored in npz format initially. The data was migrated to postgreSQL with pg-vector extension in Supabase later on. I used a relational database because there was a clear structure in my data.

Then, there was some debate on the computation required to compare the similarity of the verses. I could calculate it each time there was a search, but this would be very slow to compute with k-nearest neighbors. Because the database of bible verses would never be updated, I ended up doing the calculation beforehand and saving top 50 verses for each verse in the DB.

Next, I thought it would be useful to have a feature to compare the user query with the verses. For example, if the user searches the word "love", the app would return verses that are most semantically similar with the word. This is done through a simple brute force KNN algorithm. First, I thought it was the similarity calculation that that was the limiting factor, but it was actually the fetching of 31000 embeddings that caused the bottleneck. So, the embeddings are now fetched after the first search.

After deciding the structure of the project, I had to think about the best way to present the results in the frontend. Initially, it was a regular form with dropdown menus to choose a bible verse and a text box to input a query. But this looked way too dull.

Instead, I decided to represent each verse as a star, with all the verses representing a galaxy or universe of verses. This fit neatly with the main purpose of the site, which was to allow users to "explore" the verses.

Because this required reducing the embeddinng from 384 dimensions to 3, I began looking for dimensionality reduction techniques. To be honest, I am still not very familiar with dimensionality reduction techniques, but UMAP seemed to preserve the overall relation between embeddings better than others.

Because I wanted to get a working site as soon as possible, I didn't get to test with various parameters for UMAP. This, along with the inherent difficult of reducing the dimension more than hundredfold, led to a representation that is not always. If you search for different verses / queries, you will observe that some of them are close to each other, but many are also far away from one another, showing the limitation of the dimensionality reduction technique.

The website was originally deployed in Google Cloud. At the moment, I switched to Model, and the site will hopefully continue to be alive as long as Supabase and Modal does not change their free tier.

TLDR;

An AI language-learning app with Next.js framework to provide a conversational partner with personalized learning. A fine-tuned small-sized multilingual LLM is hosted with optimization techniques like quantization and adapter switching.

Features to Add Later

• Increasing performance with various prompting methods
• RAG to obtain relevant information from the entire chat history
• Online chatting with other users and group chat

Journal Entry

[Application]

This project came to my mind as I was studying spanish on my own. I was studying the language through a textbook, along with youtube videos and podcast for listening practice. However, because I was studying alone without a partner, I didn't have the opportunity to practice the languge in natural setting. So, why not build an AI conversational partner that can help me practice spanish more naturally?

I wanted to try a full stack framework, so I decided on Next.js with typescript. I also used graphQL as my api as I thought the schemas would have synergy with type safety. I used MongoDB this time, because the data would keep getting added and I wanted to stay flexible on what would be included in the database. With the big picture decided, I created a To-do list with detailed implementations, features to include, etc. A claude.md file was also designed for a easier AI workflow.

First was the basic components of the chatting app. This included a chat interface, message bubbles, date dividers, and side navigation. Then TTS and STT features were added for listening and speaking practice. At the moment, I used the webspeech API so the quality of the TTS is quite bad. I will probably work on a using a better API later on.

After a had the chatting app, next was deciding what information to collect from the user to provide a personalized learning experience. This included a brief introduction, language level, correction style, learning goals, and interest. (correction style was removed as the finedtuned model was not following the correction instructions well)

[AI Engineering]

Then, it was time to choose the model. My original model was Llama 3.2 1B Instruct. It was multilingual and small enough to be deployed sustainably.

After choosing the model, I tested the model with a series of conversations that a user may have. the response were too verbose and unnatural even after prompt engineering, so I decided to finetune the model. At first, I wanted to finetune the model on specific characters. For example, a user can chat with a Don Quioxte to learn spanish, or K-Drama character to learn Korean. But when I began searching for training data, I realized it would be impossible to get large amount of quality data for it.

So, instead I decided to finetune the model per langauge with LoRA. It would be done with 3 langauges: Korean, English, and Spanish. I chose English and Korean because I could check the quality of the response. I chose Spanish because I wanted to use the app myself to learn the language.

I still struggled to find multi-turn conversational data that could help with my finetuning. So after digging through kaggle and huggingface datasets with no satisfactory results, I decide to use synthetic data. I spent some time thinking of different variables to make the data more diverse. A list of various topics that users would generally be interested in, like travel or food. Other variables included conversation length (no. of messages), conversation position, and and CEFR levels. I also experimented with other variables like politeness and correction styles, but decided against it in the end.

After generating 2000 conversations per language, I used a the peft library in kaggle notebook to finetune the model. (free GPUs!) Training loss, validation loss, perplexity, and LoRA weights all showed that the model was learning well. I chose Huggingface space to host the finetuned model, as it offered 16GB of RAM for free. Once I set up the APIs for token streaming, adapter switching, and a simple gradio UI, I began testing the model.

Then I ran into 2 issues.
1. Despite having minimal system prompt and 1B paramter model, inference was extremely slow, taking 5-10 minutes to generate a simple response.
2. The performance was just bad. It was subpar in English, but struggle to understand the language or make coherent sentences in Spanish and Korean. I initially thought there was something wrong with finetuning, but the problem persisted with the base model as well. (In hindsight, I should have tested the performance before finetuning...)

To solve the first issue, I looked for ways to optimize inference on CPU. I quantized the model to Q4_K_M, tried the GGUF format and llama-cpp library, but it was still too slow. After many tries, I decided to move on to a hosting service that provided GPU and more RAM.

Because I was no longer restrained by the resources in Huggingface space, I searched for a better model. I ended up with Qwen2.5-7B-Instruct. This base model initially was giving Chinese responses to Korean prompts, and I thought about adding filters for each langauge. Fortunately, this issue was fixed with finetuning.

Finally, I designed a few different personalities for the chatbot to give more diverse experience for the learners. These included a curious personality a playful one, and more. Currently, they are done through prompt engineering, but it may change finetuning later on.

[Return to Application]

After that, it was adding the final touches, checking for errors, and deployment. Authetication was done through Auth.js with option for Google Credentials. Because my GraphQL subscription required a Pub/Sub, Redis was used. This setup would also be useful later for online group chats.

Currently, the model is hosted without a container and cold starts with requests, leading to slow intially responses. Of course, both the performance and the speed would improve by using SOTA model APIs, but I wanted to have full control of all the parts of this project.

As of March 2026, some new features have been added. An improve my sentence button has been added to help learners check their mistakes and improve upon them. A translate button has also been added for bot messages to help users understand phrases that they do not know.

To make it easier for people to try the website, I made a demo account which users can use without signing up. The mobile frontend interface has also improved and the program now goes through e2e testing.

TLDR;

An educational RAG application that allows learners to interact with study materials, create summaries, and test their knowledge through exams.

Features to Add Later

• Use advanced RAG techniques like hybrid search, small-to-big retrieval, reranking
• Allow upload of various file types, including websites, ppt, drive, and more
• Measure RAG performance including recall, precision, and RAGAS framework

Journal Entry

[Background]

This project came to mind as I began to reflect about returning to college after my military service. One of the biggest issues of using generative AI for class content is that their answers are too general. In other words, it is not specific to the terminologies or explanations provided by the teacher, which may lead to confusion during both learning and exams. To solve this problem, I wanted to create a learning AI teaching assistant program that can provide answers specific to way that class is taught by the teacher.

Originally, I wanted to create a tool that involves both the teacher and the students. Teachers would upload materials to the tool, which will then be used to provide answers to the students. Because the teacher can view all the interactions between the students and the AI, they will be able to discover what concepts that students are struggling most with. The app will also provide metrics on how students are utilizing the AI assistant and direct the subsequent classes accordingly. The teacher can also disable the AI during exam period to make sure that students don't blindly rely on the technology without deep understanding of the material.

However, even if I built the program, I had no way of testing with the usage of the software. Therefore, I decided to switch it to a program for students. The students will upload their own materials and notes for various courses. They can interact with these materials by asking questions, generating summaries, and created sample tests for exams.

[Application]

Till now, I didn't really plan the frontend in much details. This time, I used Google Stitch to design the frontend, including the main page, chat page, summarize page, and test page. Although the experience was far from customizing every detail, it provided a good basis to start the app. As with the previous project, I continued to use React and Next.js. I did want to try another framework, but the main focus of this project was on the RAG implmentation and DB.



After completing the basics of frontend, I moved on to the backend. Python was the obvious choice, coupled with fastAPI for server and llamaindex for RAG. Because a RAG application is inherently data intensive, I spent time designing my schemas before actually beginning with code. The data would be arranged in the following order, decreasing in size: Course > Section > Document > Chunk. I also thought about using a different DB, but stuck with postgresql with pgvector extension.

Cloudflare R2 was used as object storage to store all the files uploaded by the user. The postgresql DB would be the source of truth, so deleting a course or section in the DB would in turn delete all the files associated with it in R2. The files stored in the R2 is also named after the UUID for each document in the DB.

[Feature 1: Chat]

Because the ingesting and retrieval would depend on the RAG techniques that I would implement, I spent some time reflecting on it as well. The techniques I hope to implement include the following:

• Citation - providing users with details on the source of information produced by the assistant.
• Small-to-big retrieval - having smaller child chunks for search and larger parent chunks to give context.
• Hybrid search - using both embedding and text search to find the most relevant chunks for a query
• Reranking - processing the chunks once again through another model to calculate relevancy scores

When I was implementing the chunking function, I realized that Vercel only allows files uploads that are less than 4.5MB in size. Because many documents will be bigger than this, the program uploads directly to the backend only for files. While this does expose the backend api in the frontend, I couldn't figure out another way to allow large files to be uploaded.

The ingestion process is as follows. First, the document is added to the documents tables. Then, the file is added to the cloudflare R2 and chunked using llamaindex asynchronously. There is a separate chunk_embedding table in case we switch our embedding model. The chunks overlap with each other to prevent loss of context within our chunks.

After the chunks were complete, it was time to move on to the retrieval process. The basic retrieval process was simple, with postgresql handling the consine similarity calculation with HNSW-based index. However, I noticed that the structure of the answer and the citations were not consistent. The best way to get structured answer would be function calling, but I thought I would start with strict prompting + validation and see if that is sufficient first. The prompt involves rules on how the inline citations should occur, limiting answers to information provided by the chunks, and few-shot prompting with good and bad examples. Because validation requires the entire response to come, I turned off streaming for the initial llm response.

Whenever a citation is produced, its format is validated, converted into the uuid of the chunk its referencing. In the frontend, whenever the a chunk id is found, it queries the neccesary information regarding the chunk and shows it through a tooltip when clicked. If a document (and therefore the chunk) is deleted for a response, the citations simply disappear.

[Feature 2: Summarize]

Next, I move on to the second big feature of the app - summarize. The process of summarizing depends heavily on whether the token size of the documents is safely within the context limit of the model. If the document is small, one call to the llm would be enough to produce a summary. However, if the document exceeds the context limit, it would have to be divided. I decided to summarize individual chunks and then combine the summary to create a final summary. However, because the current chunks are too small (350 tokens with 60 overlap), I decide to create parents chunks that are bigger in size. These parent chunks would also be used later for small to big retrieval anyway, so it was good to implement it early.

With parent chunks in mind, the ingestion process needed to a revision as well. Now, parents chunks that do not overlap with each other are created first, followed by child chunks within each parent chunk. Currently, for documents greater than 60% of the document limit, a batch summary with 20 parent chunks each is produced, followed by a final summary. Obviously, 20 is a very arbitrary number, and there may be a lot of improvements over the batch process. Sections and chapters within documents can be analyzed to provide more meaningful divides. However, this is work to be done later.

After designing the basic implementation of the summarize feature, I began with the details. Because the purpose of generating a summary may vastly different depending on the user and the situation, I provided options such as level of detail, audience, length, focus, and more.

Also because the summary generation process involved quite a few parallel LLM calls, it would often take minutes to generate the final summary depending on the network and the provider availability. Therefore, similar to large file upload, this feature also involves a direct call to the backend api to avoid any timeouts due to limits of vercel as a host.

Even after generating summaries that are customized for the purpose of the user, the LLM may still generate summaries that are are not fully satisfactory for the user. So, a feature to edit the summaries was also added, with the changes that user wishes being taken as a text input. Because the the edit may require more or less effort depending on the gol of the change, I added two edit types. These include a structural edit where only information in the current summary is used to reformat the summary, and a content edit where additional retrieval is performed to provide incorporate information needed to update the summary. The retrieval is similar to what is used for the chat features, with the top 5 similar chunks being used as resource.

[Feature 3: Test]

Next, I moved on to the last big feature of the app. Providing users with a feature to test their knowledge through questions. Obviously, we can just simply dump the entire content of the document and ask the llm to generate questions directly, but this won't be pedagoically beneficial. It will also lack the attention to detail or provide sufficient feedback to each question to users learn.

Therefore, after some exploration on how to best generate the questions, I ended up with a 3 step approach. First, the app asks the user the purpose of the test being generated and the number of MCQs and FRQs that they want. Then, it performs the following sequential LLM calls:

1. Process the document and extract learning objectives from it, using batching if neccessary.
2. Taking the purpose of the test into account, pick top N objectives for MCQs and top M objectives for FRQs
3. For each objective, retrieve the most relevant chunks and use that information to generate a question.

For me, this approach seemed to hit the balance between pedagogical details and resources (both time and LLM calls). It also takes advantage of the retrieval system that we already have in place. For each answer option in an MCQ question, an explanation to why its wrong/correct is generated. For each FRQ question, an ideal answer and a rubric is generated to be used for grading later on.

After that, it was testing different prompts and results and adding small features that would be beneficial. For example, I noticed that the learning objectives that was produced involved asking details about specific examples that were not very important, so the prompt was update to focus on the bigger picture. The LLM also had the tendency to generate the correct answer for the MCQ as the first or second option, so now the order of the options is shuffled each time.

Then, I thought that it would also be beneficial to allow users to generate more questions for a test, as the LLM often generated more learning objectives than was used for one question set. After finishing a test, the user has the option to either try the test again or generate more questions using the yet unused learning objectives. All the previous attempts for a test is also recording and available for review.

Now, it was time to deploy the app in production. As before, python backend is hosted using modal, and the next.js frontend using vercel. After deployment, I realized that the two process of uploading document to backend and then uploading the doucment to R2 from backend takes too long, causing time out errors. Therefore, I changed the procedure to first allow browser to upload directly to the R2, followed by backend processing the document from the R2 and producing chunks.

For now, the project has come to a temporary end. Obviously, to make it production level, I need to create authentication as well. However, this feature was already practiced in my previous project Language Buddy, and I felt no need to implement the same thing again.

Throughout this project, I think the biggest gain was experience creating a data-intensive application. I think this is the first time that I used tables and object storage to this extent, constantly thinking of the best schema to represent the data that invovled in the RAG application.

TLDR;

A python program to help students pay attention to video lectures by detecting face presence and closed eyes with OpenCV library.

Features to Add Later

• Migration to web
• In-depth analysis of student behavior to discover parts of lecture that are not engaging

Journal Entry

This project was part of the undergraduate intern research project in the AIEd Lab. With many of the courses turning remote, we knew that students often fail to engage with the material through recorded lectures - many would simply let the video play with doing various off-tasks. Therefore, we wanted to find a way that helps student focus while not being overly instrusive.

Many approaches were discussed to measure student attention, but in order to keep the the project simple, we opted for face presence and open eyes as a marker for student paying attention to the lecture. The lecture would stop if face was not detected or eyes were closed for a certain period of time.

After developing a prototype, we ran tests on lab computers. Realzing that the program would not run smoothly on many of the computers, we switched to multithreading to have the video and face detection run parallel.

Unfortunately, I had to join the military service before the experiments actually took place.