CoLoop is a new ‘AI copilot’ for qualitative analysis, developed by Genie Technology Inc., and is one of several new generative-AI tools coming onto the market specifically designed to facilitate qualitative analysis of text. For a high-level summary of CoLoop’s functionality, see my previous post. Here I share some reflections from my initial experimentations with it, from setting up a project to going through the AI-generated analysis options currently available.
Note: at the time of writing CoLoop was in pre-release beta. The Ap is being
developed continually so functionality may have changed by the time you are reading
this – so be sure to check out their website for latest updates.
Getting started
Getting going is really easy, you log into your account, create a new ‘project’ for your qualitative analysis, and upload the texts. At the moment text needs to be formatted as .txt format – a straightforward process to save into this format from your standard word processor, but an additional preparatory step if you already have transcripts prepared. Hopefully in the future they will enable other text formats to be uploaded, including PDFs which many researchers would find useful.
Automated transcription in CoLoop
Alternatively you can use CoLoop’s automated transcription tool – and upload audio or video (mp3 or mp4) and have it automatically transcribed. I tested this twice:
with a short audio file of me speaking, it was near-perfect – transcribing talk into sentences as accurately as I would have done if manually transcribing, in all bar one instance. And only making an ‘error’ on a technical term (the name of a software program) that is a very minor issue and easily corrected.
with two hour-long podcast conversations, again it was very accurate – one of them almost perfect (except for technical terms such as CAQDAS which it heard as cactus (nothing surprising there!!).
I was impressed with this – there’ll be a post in the future comparing automated transcription tools…particularly those embedded within CAQDAS-packages, which as I mentioned in the previous post is a current development in several tools.
Data type assumptions
CoLoop is built around the concept of enabling the qualitative analysis of transcripts of focus-group discussions (FGDs) and individual interviews (IDIs). Therefore after automatically transcribing your audio or video file – or uploading a pre-transcribed document CoLoop looks for speakers, and automatically identifies them if you have used consistent unique identifiers for individuals (or where it hears different voices when automatically transcribing). Having identified the speakers, it asks the user to specify who is the ‘Researcher’ and who is the ‘Participant’ in a transcript. In the first audio I uploaded there was only one speaker, and because it wasn’t the transcript of an interview or focus-group that option request didn’t quite make sense. For my podcast episodes, identifying the named speakers was useful and accurate; and even though assigning who was ‘researcher’ and who was ‘participant’ didn’t really make sense for this data, it wasn’t a big problem (although it would be great to be able to user-specify role types for non-FG or IDI data).
Auto-summarisation
Having uploaded transcriptions in one of the above ways, CoLoop automatically indexes each transcript into its AI memory – which allows it to auto-summarise the transcript. When there are two speakers only, and one has the identifier ‘Interviewer’ CoLoop will automatically identify that the ‘Interviewer’ is the ‘Researcher’ and the other speakers are the ‘Participants’. This worked a bit differently with the two sets of data I experimented with (the conversations that were automatically transcribed and some interview transcripts), which seems to be related to the length of each speaker section.
with interview transcripts where there is a lot of prompting and probing from the interviewer the summary is not much less in length than the original full transcript, so the summary wasn’t of much additional benefit (although with other types of interview or focus group transcripts it may well be useful)
with the transcripts of podcast discussions, where there is a lot more talk from one speaker than the other, the auto summary reduced the length of the transcript dramatically (e.g. in one from 8800 words (13 ½ pages) to 2200 words (3 ¾ pages). I found the summary useful as an overview – something I can envisage sharing with colleagues as the basis of an initial discussion about the material, before analysis commences on the full transcripts.
BELOW: Screenshot showing auto-summarisation of a transcript in CoLoop (taken from episode 4 of my #CAQDASchat with Christina podcast where I chat with Dr Stu Shulman about humans and computers working together in qualitative data analysis)
You’re now ready to use the technology to ask questions of the data. This can be done in two ways: via the Analysis Grid or the Chat function….
The Analysis Grid
The analysis grid is a good starting point for analysis because in this space you can add additional ‘segments’ to the participants – which could be socio-demographic characteristics – pertaining to each individual. This allows subsequent retrieval to be sought in relation to the characteristics represented by the segments.
You can then start asking questions of the data, and CoLoop searches the transcripts material that answers your questions which are displayed in columns, such that the findings for each transcript are displayed in a separate cell (see figure below).
BELOW: screenshot showing the Analysis Grid in CoLoop (columns of questions, rows of participants - cells with AI-generated answers to prompts)
The evidence from which these summaries (which may be presented in paragraphs of bullet lists depending on how you ask the questions) can be accessed directly from the summaries by clicking on the summary sentences – which displays the extracts on which the summaries are based within their transcript context.
Chatting with CoLoop
The other way to question your data using CoLoop is to use the Chat function – which looks and feels a lot like what you might be used to if you’ve experimented with ChatGPT and similar other generative-AI tools. You basically pose a series of questions in a conversation-like stream and the extracts upon which the answers are based again can be directly accessed (as shown below). You can delve deeper and deeper into the data by asking follow-up questions.
BELOW: The CoLoop Chat interface where you can ask questions of your qualitative texts
So what’s it good for?
I’m pretty impressed with CoLoop and can see how it might be used by qualitative researchers for certain purposes. Because it generates summaries directly from the data, if your need is to quickly understand what is contained within textual material, focused around specific content and in relation to characteristics you can capture with the ‘segments’ feature then this is a powerful way of familiarisation.
The results though are based on how you ask questions, so you need to experiment with writing good prompts. Luckily, when you have a CoLoop account you can access advice on writing good prompts to ensure the most useful summaries are generated from your data.
Although CoLoop is pretty good at generating summaries based on segments, what I look forward to is developments that allow the material to be tagged such that patterns and relationships beyond descriptive summaries can be accessed and interrogated in further detail. By this I mean the sort of qualitative coding tools we see in established CAQDAS packages.
I can clearly see how those whose aim is to summarise qualitative material will be attracted by CoLoop, especially when the amount of time to turn an analysis round is limited. But those qualitative researchers who are embedded in the interpretive methodological space, who seek to understand nuance in-depth and are interested in conceptual connections and relationships, need to do more with their data than generate summaries – however ‘accurate’ they may be descriptively. Therefore, at the moment, for many academic and applied qualitative researchers CoLoop may be an additional tool to explore to augment their analytic process rather than replace their established procedures (whether manual, paper-and-pen methods or the use of other CAQDAS packages).
That said, CoLoop is developing very quickly – the added features as I was trialling and are very open to feedback from researchers experimenting with the platform, so there are great opportunities to contributing to the development of this tool.
Make informed decisions: try it out for yourself
It's always best to make your own decisions about what tools work best for you. It's great to hear other researcher's thoughts and experiences, and hopefully this post is useful in that respect, but your project is yours, you are the expert in what is needed, whether its data, methods or tools. The best way to decide whether a new development is appropriate is to try it out for yourself. That's the only way any of us can make truly informed decisions
コメント