What Is Being Said? Analyzing Presidential Debate Transcripts

[Francis_Kabinoff]

Have you watched any of the debates so far this year? How about the town halls last night? We thought it would be fun to do some debate analysis using Qlik Sense, so we did some basic natural language processing on debate transcripts dating back to 1984, loaded the data into Qlik Sense, and created an app to explore what's being said. You can check it out @ https://qdt-apps.qlik.com/presidential-election-2020/. Let's take a look at how we did it.

First, we had to generate some data from the transcripts. Obviously, we wanted who's saying what words, but we also wanted to have some interesting ways to explore the data, so we decided to include topics, parts of speech, and sentiment. We also wanted to include not just single words but bigrams (two-word phrases) and trigrams (three-word phrases). We wrote a python script which utilized NLTK (https://www.nltk.org/) to generate the data and wrote it to CSV files, which we then loaded into a Qlik Sense app.

With our data now in a Qlik Sense app, we used the Word cloud chart that now comes with Qlik Sense in the Qlik Visualization bundle to create a word cloud for each party using set analysis. We then added the fields we wanted to filter on to a filter pane, and were quickly and easily able to start exploring our word clouds.

Then we used the Capability APIs to embed the filters and word clouds into a web app. We thought it would add a bit of fun to create checkbox filters for the "Sentiment" and "Words in Phrase" fields, and if you're interested in how to build checkbox filters like that yourself, I've included the code at the end of this blog post.

And that's it. Again you can find the completed project @ https://qdt-apps.qlik.com/presidential-election-2020/. Play around with it, have some fun, and let me know if you discover anything that's really interesting to you.

Here's the code for the custom filter checkbox component, in case you're interested.

import React, {
  useCallback, useRef, useState, useEffect,
} from 'react';
import PropTypes from 'prop-types';
import { makeStyles } from '@material-ui/core/styles';
import Checkbox from '@material-ui/core/Checkbox';
import { appPromise } from '../config/app';

const useStyles = makeStyles({
  root: {
    display: 'flex',
    alignItems: 'center',
    justifyContent: 'space-between',
    border: '1px solid #ccc',
    padding: 5,
    borderRadius: 3,
  },
  checks: {
    display: 'flex',
    alignItems: 'center',
    justifyContent: 'center',
  },
  legend: {
    marginRight: 5,
    fontWeight: '700',
    fontFamily: '"QlikView Sans"',
    fontSize: 13,
    color: '#595959',
    whiteSpace: 'nowrap',
  },
  label: {
    display: 'flex',
    alignItems: 'center',
    marginRight: 5,
    height: 20,
  },
});

const useCheckboxStyles = makeStyles({
  root: {
    padding: 0,
  },
});

const CheckboxFilter = ({ field, labels, hideLegend }) => {
  const classes = useStyles();
  const checkboxClasses = useCheckboxStyles();
  const app = useRef(null);
  const model = useRef(null);
  const [layout, setLayout] = useState(null);
  useEffect(() => {
    (async () => {
      app.current = await appPromise;
      model.current = await app.current.createSessionObject({
        qInfo: { qType: 'field' },
        qListObjectDef: {
          qDef: { qFieldDefs: [`[${field}]`] },
          qInitialDataFetch: [{
            qWidth: 1,
            qHeight: 100,
          }],
        },
      });
      model.current.on('changed', async () => { setLayout(await model.current.getLayout()); });
      setLayout(await model.current.getLayout());
    })();
  }, []);

  const handleChange = useCallback((event) => {
    const qElemNumber = Number(event.target.name);
    model.current.selectListObjectValues('/qListObjectDef', [qElemNumber], true);
  }, []);

  return (
    <div className={classes.root}>
      <div className={classes.legend}>
        {field}
      </div>
      <div className={classes.checks}>
        {layout && layout.qListObject.qDataPages[0].qMatrix.map((row) => (
          <label
            key={row[0].qElemNumber}
            htmlFor={`${field}${row[0].qElemNumber}`}
            className={classes.label}
          >
            {labels && labels.find((label) => label.qText === row[0].qText).component}
            {!labels && row[0].qText}
            <Checkbox
              checked={row[0].qState === 'S'}
              onChange={handleChange}
              name={String(row[0].qElemNumber)}
              id={`${field}${row[0].qElemNumber}`}
              classes={checkboxClasses}
            />
          </label>
        ))}
      </div>
    </div>
  );
};

CheckboxFilter.propTypes = {
  field: PropTypes.string.isRequired,
  labels: PropTypes.array,
};

CheckboxFilter.defaultProps = {
  labels: null,
};

export default CheckboxFilter;