From Manual Nightmare to AI Automation: Fixing 2,000 Code Samples

2 hours ago8 min read

Wix’s Velo API reference contains over 2,000 code samples. About a year ago, we introduced a syntax change to Velo that suddenly rendered a huge number of these samples outdated. Updating the reference manually would have required an enormous amount of tech writer work. Our team decided to try something new and automate this process using AI.

Here’s our journey.

The change

The syntax change involved the way that web methods are exported from backend code to frontend code using Velo. Originally, we exported web methods from files with a proprietary .jsw extension. Any function exported from these files could be imported into frontend code.

The code looked like this:


import { members } from "wix-members.v2";
export async function myGetMemberFunction(_id, options) {
	try {
		const member = await members.getMember(_id, options);
		console.log("Member retrieved:", member);

		return member;
	} catch (error) {
	  console.error(error);
	  // Handle the error
	}
}

In the new syntax, web methods are exported from files with a .web.js extension. The exported functions must also be wrapped with a webMethod function.

The new code looks like this:

import { members } from "wix-members.v2";
import { webMethod, Permissions } from "wix-web-module";
export const myGetMemberFunction = webMethod( Permissions.Anyone, async (_id, options) => {
	try {
		const member = await members.getMember(_id, options);
		console.log("Member retrieved:", member);
	return member;
  } catch (error) {
	console.error(error);
	// Handle the error
	}
  },
);

Framing the problem

Before starting to work, our team considered the problem in detail. We realized that several factors made automating this project more complex:

The syntax change only affected code samples that demonstrated exporting and importing web methods. Many of our samples don’t include this. We had no programmatic way of knowing which specific samples we needed to change.
The style guide for writing Velo code samples evolved over several years. This meant that the code samples were in a variety of formats. We couldn’t expect them all to have the same structure.
The tech writing team at Wix works in a docs-as-code system. We maintain some source files for the docs in our own GitHub repo. Other files are dispersed among Wix’s many repos that also house the source code for the APIs. There was no straightforward way (that we could find) to identify the source file for any given code sample.

Planning a solution

Once the complexities of the project were clear, we came up with an action plan.

Code classification

We decided to use an LLM to classify our code samples and figure out which ones we needed to convert. The probabilistic nature of LLMs meant that some percentage of the classifications would almost certainly be wrong. We reasoned that it might not be worse than human error if we could write a prompt that performed well on a testing data set.

Here’s the prompt we came up with:

You are a bot who is analyzing Velo by Wix code samples. Your task is to identify backend code samples that need to be rewritten because of a change to the Velo syntax. The new syntax defines a new way to export functions from backend code.

Here are the rules for something to be considered a relevant backend code sample:

+ They MUST include an import statement that imports from a package with either 'backend' or '.v1', '.v2' (or a higher version) in the name.

+ They MUST also include an exported function which is usually asynchronous.
 
+ The following cases are not relevant backend code samples EVEN THOUGH they include an exported function:
+ Event handlers (definition below). Do NOT consider event handler functions as backend code.

+ HTTP functions (definition below). Do NOT consider HTTP functions as backend code.

+ Some code samples include both backend and frontend code. These samples include a comment line that includes "Backend Code". These samples ARE considered relevant backend samples.

Definitions:

Event handlers. These samples include the following details:

+ They export a function with the following signature pattern: `export function {service}_on{event name}(event)`. For example: `export function wixBlog_onCategoryUpdated(event)`.

+ The function is not asynchronous.
HTTP functions. These samples include the following details:

+ They export a function with the following signature pattern: `export function <CRUD method>_<function name>(request)`. For example: `export function get_muffinDetails(request)`
The user will provide only the code sample. 

You must respond only "yes" if the code sample is a backend code sample as defined above or "no" if it isn't.

We used this prompt as the system message. The user message was only the code sample to classify. We chose the GPT-4 model and tested the prompt using a set of code samples. It failed only a small number of times.

Code conversion

The fact that the code samples were in different and unpredictable formats meant that we couldn’t easily script a way to convert the code from the old syntax to the new one. Because of this, we decided to use an LLM to convert the code as well.

Here’s the prompt we wrote for this task:

System Message:You are a professional JavaScript software developer who is also a technical writer. You provide code examples for technical documentation.

User Message:

We need to rewrite the backend code examples (and only the backend code examples) that appear in the reference documentation for Wix's Velo JavaScript development platform.

Backend functions are not run in the browser, but rather on the server, which allows for greater security. They are defined in backend code files, and they are imported in Velo frontend code. The old style were called backend functions. The new kind are called web methods.

The format of these backend functions has now changed, and we need our code examples to reflect this. I will explain the things that have changed. Then I will give you some examples of old code examples and how they look now. Then I will give you a code example of the old kind and your response will be the same code example converted into the new format.

Explanation of the change:

In the past, Velo backend functions were written in backend code files with names like `my-backend-file.jsw`. In these files, the backend function would be exported as follows:

`export {async} function myFunc (params){...}` (for example: `export async function myAddToCartFunction(_id, options) {...}`).
If the code example also includes frontend code importing the function, it would look like this: `import { myFunc } from 'backend/my-backend-file'`

The new version includes ONLY the following changes:
+ Backend code should always include the following import statement:

`import { Permissions, webMethod } from "wix-web-module";`

+ Function definitions should now be written according to the following pattern:
`export const myFunc = webMethod(Permissions.Anyone, {async} (params) => {...});`

+ Instead of backend code file names like `my-backend-file.jsw`, the new backend functions are in Web Method files are named with this structure: `my-backend-file.web.js`.

To convert a code example, only the above elements need to be changed. Everything else should be left exactly as it is.
For example:
OLD:
```
import { activityCounters } from 'wix-activity-counters.v2';
	async function getActivityCounters(memberId) {
		try {
			const result = await
		activityCounters.getActivityCounters(memberId);
		
		return result;
		} catch (error) {
		console.error(error);
		// Handle the error
		}
	}
```
NEW:
```
import { Permissions, webMethod } from 'wix-web-module';
import { activityCounters } from 'wix-activity-counters.v2';
	export const getActivityCounters = webMethod(Permissions.Anyone, async (memberId) => {
	try {
		const result = await
		activityCounters.getActivityCounters(memberId);
		return result;
		} catch (error) {
		console.error(error);
		// Handle the error
		}
	});
```

Below is the example in the old style I would like you to convert to the new style.
Do not make ANY changes other than those mentioned and shown above. 

If the code example contains comments including sample values or responses, include those as they are.

```
< code sample to convert >
```

After some experimentation we found that GPT-3.5 turbo produced the best results out of the models we had available at the time.

Output validation

Converting the code samples with AI meant that a certain percentage of the conversions might be wrong. We couldn’t publish bad code in the reference. So in this case we needed to minimize the amount of errors from the LLM output as much as possible.

To validate the converted samples, we used a utility that retrieves the type definitions for all the Velo APIs and uses them to compile code samples using the TypeScript compiler. By compiling each converted code sample, we could at least confirm that the new code was valid JavaScript and used the Velo APIs correctly.

We decided to manually review all validations errors and fix them as needed.In practice, we never saw a single example of the LLM failing to convert the code properly. The only errors we encountered were cut-offs where the LLM hit the return token limit before completing the sample. We also found a few cases where the samples failed to compile because the type definitions were out of sync with the underlying APIs.

Search and replace

To find and replace the source files, we made use of GitHub’s APIs using the GitHub CLI. Each Velo API has a unique identifier and the file structure for the sample files is consistent in every source repo. We wrote a script that searches Wix’s GitHub repos for an API’s ID and then retrieves all the code sample files for that API.

To replace the files after rewriting them we wrote another script that cloned the relevant source repo locally, made the necessary changes, and then created a PR.

Implementation

To implement our plan, we created a new repo to host our scripts and the code sample files to collect and convert. We followed this workflow:

Search: Our script used the GitHub CLI to search for code sample files throughout Wix’s source repos and save them in our working repo. The script also created a mapping file to correlate the location of the sample files in our repo and the source repo.
Classify: Another script used GPT to identify which code samples required rewriting for the new syntax. We used the prompt detailed above. We accessed GPT via API and dynamically inserted the code sample into the prompt for each request. The script added a flag for the files that required rewriting to the mapping file.
Convert and validate: Next, a script sent each file that required rewriting to GPT with our conversion prompt. We stored the rewritten files near the originals in the repo for easy manual comparison. The script also used our validator to see if the new code would compile. We saved any validation errors in a log file together with original and converted code samples. The script also compiled a master list of all the converted samples that returned errors.
Handle errors: We divided the validation among a team of writers. For each error, the writer checked if it resulted from the code conversion and applied any necessary fixes. Once we merged all the fixes to the repo, we moved on to the next phase.
Replace: Our final script used the mapping file to apply all our code sample conversions to their respective source repos and returned a list of PRs. From here, tech writers could merge and publish the updated docs.

Results

Manually locating, classifying, and rewriting each code sample would have taken a huge amount of work and brought our whole team’s general velocity to a halt. By automating the process, we were able to limit team involvement to 6 writers who worked on the project for one morning. After that, it was just a matter of merging PRs and publishing the docs.

We did have to make an upfront investment in planning and development. However, this involved far fewer members of the team, which allowed the team overall to continue its usual work. The knowledge we gained and the code we wrote have since helped us to complete similar tasks much faster.

Conclusion

Tedious, repetitive tasks like converting code sample syntax can be automated, saving many hours of tech writer work. Even if your team doesn’t have the in-house coding skills to develop a solution like ours, it may be worth it to request developer resources for this.

This can free up your team to work on more sophisticated content that can’t be easily automated and usually provides greater value to your users.

This post was written by Adam Friedmann

More of Wix Engineering's updates and insights:

Follow us on: Twitter | Facebook | LinkedIn
Join our Telegram channel
Visit us on GitHub
Subscribe to our monthly newsletter
Subscribe to our YouTube channel
Follow our Medium publication
Listen to our podcast on Apple or Spotify