diff --git a/source/images/qualitative_analysis/add_question.png b/source/images/qualitative_analysis/add_question.png new file mode 100644 index 000000000..660e82d70 Binary files /dev/null and b/source/images/qualitative_analysis/add_question.png differ diff --git a/source/images/qualitative_analysis/data_table.png b/source/images/qualitative_analysis/data_table.png new file mode 100644 index 000000000..699fa5969 Binary files /dev/null and b/source/images/qualitative_analysis/data_table.png differ diff --git a/source/images/qualitative_analysis/display.png b/source/images/qualitative_analysis/display.png new file mode 100644 index 000000000..5033849ad Binary files /dev/null and b/source/images/qualitative_analysis/display.png differ diff --git a/source/images/qualitative_analysis/generate.png b/source/images/qualitative_analysis/generate.png new file mode 100644 index 000000000..f251e7228 Binary files /dev/null and b/source/images/qualitative_analysis/generate.png differ diff --git a/source/images/qualitative_analysis/hints.png b/source/images/qualitative_analysis/hints.png new file mode 100644 index 000000000..473e80e42 Binary files /dev/null and b/source/images/qualitative_analysis/hints.png differ diff --git a/source/images/qualitative_analysis/switch_question.png b/source/images/qualitative_analysis/switch_question.png new file mode 100644 index 000000000..b9d957d69 Binary files /dev/null and b/source/images/qualitative_analysis/switch_question.png differ diff --git a/source/images/qualitative_analysis/switch_submission.png b/source/images/qualitative_analysis/switch_submission.png new file mode 100644 index 000000000..278edae50 Binary files /dev/null and b/source/images/qualitative_analysis/switch_submission.png differ diff --git a/source/images/qualitative_analysis/verify.png b/source/images/qualitative_analysis/verify.png new file mode 100644 index 000000000..b7e1a3760 Binary files /dev/null and b/source/images/qualitative_analysis/verify.png differ diff --git a/source/qualitative_analysis.md b/source/qualitative_analysis.md index f98164733..a408e3185 100644 --- a/source/qualitative_analysis.md +++ b/source/qualitative_analysis.md @@ -1,59 +1,153 @@ -# Qualitative Analysis of Audio Responses +# Qualitative analysis of audio responses in KoboToolbox **Last updated:** 3 May 2024 + -Qualitative analysis can be crucial for research and emergency response, offering in-depth insights often missed in quantitative data collection. KoboToolbox’s qualitative analysis tools help users to efficiently process and understand complex information from open-ended audio responses. Using the qualitative analysis features, you can add new data fields for summarizing, categorizing, and describing audio responses. +Qualitative analysis helps you turn open-ended audio responses into clear, usable insights. This is especially valuable in research and emergency response, where important context can be missed in quantitative data alone. -The first step is to [collect audio responses](photo_audio_video_file.md) in your form, using the Audio question type. It may be helpful to [transcribe and/or translate](transcription-translation.md) your audio files before you begin the analysis, but it is not required. +KoboToolbox offers two ways to analyze qualitative audio responses. You can either **analyze data manually** or use **artificial intelligence (AI)** to generate the analysis. Using analysis questions, you can summarize, categorize, and describe each response, then save those results as new columns in your dataset. -

The qualitative analysis features are currently only available for audio responses and cannot be used for background audio recordings or other question types.

+This article explains how to create analysis questions, analyze responses manually or with AI, review and verify results, customize display settings, and move between responses during analysis. -## Start analyzing your qualitative data +## Creating analysis questions -1. Open your project and navigate to **DATA>Table**. +To use qualitative analysis, your form must include at least one **Audio** question or have [background audio recording](https://support.kobotoolbox.org/recording-interviews.html#recording-interviews-with-background-audio-recordings) enabled. It may be helpful to transcribe or translate your audio files before you begin, but this is not required. -2. Click the **Open** button next to the audio response you would like to analyze. +

+Note: + Qualitative analysis is currently available only for audio responses, including background audio recordings. It is not supported for text or other response types. +

-3. Open the **ANALYSIS** tab to get started. +To start analyzing your qualitative data: -![image](/images/qualitative_analysis/Open-analysis.gif) +1. Open your project and go to **DATA > Table.** +2. Click **Open** in the cell of the audio response you want to analyze. +3. Open the **ANALYSIS** tab. +4. Click **Add question.** +5. Enter a title for the analysis question. + - This title becomes the column name in your dataset. +6. Select the question type you want to use, and add answer choices if needed. -## Change your display settings +![Add question](images/qualitative_analysis/add_question.png) -By default, the display panel on the right side of the screen will show the recording and original transcript of your audio file. You can change the display to include the information that will support your analysis. For example, if you are working in multiple languages, you can change the display to include translations or to remove the original transcript. +Each analysis question you create will appear in the **ANALYSIS** tab for other responses to the same audio question. -1. To change the display, click **Display settings** and select the information you would like to display. +

+Note: +You can also add hints to analysis questions or answer choices. This is a good place to add information from a codebook or instructions for coders. +

-2. You can choose to show or hide the audio recording, submission data (i.e., participant’s responses to other questions from the form), original transcript, and any translations of the transcript. +### Analysis question types -

If you haven’t transcribed your recording, only the audio recording and submission data will appear on this display.

+The following question types are available for analysis questions: -![image](/images/qualitative_analysis/Display-settings.gif) +| Question type | Description | +|:----|:----| +| Tags | Add keywords or themes to describe the audio response. | +| Text | Add an open text response, such as a summary, notes, or overall impression. | +| Number | Record a number, such as the respondent's age or the number of times a theme is mentioned. | +| Single choice | Select one option from a list, such as the main theme or perceived level of satisfaction. | +| Multiple choice | Select one or more options from a list, such as themes or barriers mentioned in the response. | +| Note | Add instructions or section labels to organize the analysis workspace. | -## Using the analysis features +Each field you add becomes a new column in your dataset when you download your data. -The data analysis features allow you to create different types of data assessment fields to describe and categorize your audio responses. The fields you add will be included in the **ANALYSIS** tab for all other audio files on the form. To add a field, click the **Add question** button and enter a title. The title will become the column name for this field in your dataset. +### Adding hints to analysis questions -The following types of data assessment fields can be used: +Hints can help make your analysis more consistent, whether responses are reviewed by human coders or generated with AI. Use hints to explain how each analysis question should be answered. -- **Tags:** Add keywords or themes to qualify and describe your audio file. -- **Text:** Add an open textbox to describe your audio file (e.g., summary of response, notes from interviewer, overall impression). -- **Number:** Add a number to describe your audio file (e.g., number of times a theme is mentioned, age of respondent). -- **Single choice:** Select one from a list of options to describe your audio file (e.g., tone of voice, level of satisfaction). -- **Multiple choice:** Select one or multiple from a list of options to describe your audio file (e.g., barriers to healthcare mentioned by respondent). -- **Note:** Add a note to organize your analysis fields or leave instructions to others. Notes will also appear in the ANALYSIS tab for all responses collected for this question on the form. +For example, you can add: -![image](/images/qualitative_analysis/Analyze.gif) +- A full codebook or coding framework +- Definitions for tags, categories, or themes +- Examples of how to apply each answer choice +- Instructions for handling unclear or incomplete responses +- Any guidance you would normally give to a human coder +- Prompt-style instructions for AI-generated analysis -Each field you add will become a new column in your dataset, which you can download with the rest of your data. +![Add hints](images/qualitative_analysis/hints.png) -## Analyze multiple audio files +Hints can be **especially useful when using AI**, because they give the AI more context about how to interpret the audio response and how the analysis should be structured. -You can only analyze one audio response at a time, but it is easy to switch from one question or respondent to another. In the **ANALYSIS** tab for each audio file, the fields you have added for any audio file will appear, ready to be filled in. +Hints do not have a word limit, so you can include detailed instructions when needed. We recommend keeping hints clear and specific so they are easy for both team members and AI tools to follow. -- To switch to the next data entry, click the arrows to the left of the **DONE** button. -- To switch to another question within a single data entry, click on the drop-down menu at the top the screen, and select the question you would like to analyze. +

+Note: + If your hints are very long, such as detailed instructions for AI-generated responses, you can disable the Show hints button at the top of the Analysis window to hide them. +

+ +## Analyzing your data + +You can analyze responses manually or use AI to generate a response. + +- **For manual analysis:** Manually enter a response for each analysis question. +- **For automated analysis:** Click Generate with AI under the question. After a few seconds, a response will be generated. You can then review the response and edit it if needed. + +![Generate analysis with AI](images/qualitative_analysis/generate.png) + +

+Note: +A response that has been generated with AI will include the mention AI-generated underneath the question. +

+ +### Reviewing and verifying responses + +For both manual and AI-generated analysis, you can review each response and mark it as verified. This can help with quality control, whether you are checking coding across a team or confirming that an AI-generated response is accurate. + +To verify a response, check the **Verified** box below it. If you leave the box unchecked, the analysis result will still be saved, but your team will be able to see that it has not yet been reviewed. + +![Verify analysis](images/qualitative_analysis/verify.png) + +### Viewing and exporting analysis data + +When you finish analyzing your audio files, each analysis field is saved as a new column in your dataset. Your dataset will also include a **Verified** column with **Yes** or **No** values. + +![Data table](images/qualitative_analysis/data_table.png) + +You can [download](https://support.kobotoolbox.org/export_download.html) your data with these analysis fields included for further review, synthesis, or reporting. For example, you can use them to track how often specific themes appear across your transcripts. + +### Customizing the display settings + +By default, the display panel on the right side of the **ANALYSIS** screen shows the audio recording, the original transcript, and responses to other questions. + +You can change the display to include the information that best supports your analysis. For example, if you are working in multiple languages, you may want to show a translation or hide the original transcript. + +To change the display: + +1. Click **Display settings** in the top right corner. +2. Select the information you want to show. + +![Display settings](images/qualitative_analysis/display.png) + +You can choose to show or hide: + +- The audio recording +- Responses to other questions in the form (all or specific questions) +- The original transcript +- Transcript translations + +If the recording has not been transcribed, only the audio recording and submission data will be available. + +

+Note: +To learn more about transcription and translation of audio responses in KoboToolbox, see Transcription and translation of audio responses. +

+ +### Switching to a different question or transcript + +You can analyze only one audio response at a time, but you can move easily between responses and questions. + +To switch to the next or previous submission, use the arrows to the left of the **DONE** button. + +![Switch submission](images/qualitative_analysis/switch_submission.png) + +To switch to a different audio question within the same submission, use the drop-down menu at the top of the screen and select the question you want to analyze. You will be able to add **new analysis questions** for this audio question. + +![Switch question](images/qualitative_analysis/switch_question.png) + +## Usage limits for AI-generated analysis + +Community Plan users can make up to 25 AI-generated analysis requests for free. Each time you click **Generate with AI**, it counts as one request. + +If you need more AI-generated analysis requests, you can [upgrade](https://www.kobotoolbox.org/pricing/) to a plan with a higher quota or purchase a **Natural Language Processing (NLP)** add-on. ​​You can always continue using the manual analysis features with no usage limit. -![image](/images/qualitative_analysis/Change-response.gif) - -When you have finished analyzing your audio files, each data assessment field will be included in your dataset as a new column. You can then download and process the dataset, with these new fields included and ready for further analysis and synthesis.