Guide sharing site "JMP Public"
In JMP, analytical reports can be shared on the website JMP Public We provide a site called.
Latest version of JMP released at the end of 2018 In JMP 14.2, you can upload interactive HTML analysis report to this JMP Public by selecting [File] → [Publish] from the report window.
You can set the scope of the report so that only the principal who uploaded, or all people can view it (You will need to sign in to the JMP Public site using the SAS profile for uploading).
A lot of analysis reports are uploaded to JMP Public.
Since there is a report submitted by the engineer of JMP Japan Division, please enjoy it along with articles detailing the contents of that report.
|
When text mining the music of the band "Queen"
Movie released at the end of last year "Bohemian Love Sody" Have you seen it?
It seems that a wide range of generations are being watched from the generation heard during the period when Queen was active, to the young person who does not know much about Queen itself, as the fun of the movie became a hot topic immediately after the release.
Although I also appreciate it, I was struck by the songs that I can not fade even listening now, while also listening to the story.After that, from the thought of wanting to know more about Queen's poetry, suddenly (?I tried text mining.
Before analyzing, I knew only the representative song of Queen, but I called a word "Bicycle" and called "Killed a man" like a song "Bohemian Rhapsody" Because there are lyrics and characteristic lyrics of the opera part, I thought that something interesting results could be obtained.
We will introduce the following three results from text mining.
- What kind of words are used a lot?
- What if I try to sort the songs by the words I use?
- Can you distinguish lyrics from words contained in lyrics?
The subjects of analysis were songs recorded in the album from 1973 to 1991, and songs (instrumental) without lyrics were excluded.It was not until 1991 because I wanted vocal Freddie Mercury to target songs that were released during their lifetime.
What kind of words are used a lot?
What kind of words are used more frequently for the songs to be analyzed?The answer is shown in the word cloud.The word cloud is a visualization method often used in text mining, and displays it with larger letters as the frequency of appearance of words increases.
Word cloud (word appearance frequency)
Please note the following three points about the word cloud displayed by clicking above.
We perform word stem extraction and display words with a frequency of 10 or more.
Extracting word stem is a way to eliminate the change in the ending of a word and put together words with common stems.The word "stem" is marked with "·" at the end of the word.For example, "know ·" is displayed in the word cloud, but it contains three words "know", "knowing", "knows".
- We exclude words that are considered articles, auxiliary verbs, pronouns, be verbs, prepositions.
- For words, we have a gradation with blue (average release year early) to red (average release year late).In other words, a blue color word is a word used more often in early songs, and a red color word indicates that it is a word used more in later songs (as of the end of 1991).
What if I try to sort the songs by the words I use?
Let's classify (group) the target songs into songs that use similar words.To that end, first, Document word matrix I will introduce the concept.
The document word matrix is a matrix representation of whether each word appears or not.A row shows music and each column shows each word. For a word string, assign a value of 1 to the song (line) in which the word appears and assign a value of 0 to the musical piece that does not appear I will.Although it is simple to create a document word matrix that takes a value of 1 or 0 like this, here we create a document word matrix weighted with the importance of the word "TF IDF" taken into account .
Although detailed definitions are omitted, we will lower the importance of words appearing in many songs and weigh them to increase the importance of words appearing only in some songs.
Consider two examples of "life ·" (life, life's), "bicycl ·" (bicycle, bicycling) extracted as a word stem.
The table below is part of the weighted document word matrix.
Where a value other than 0 is contained, it means that the word (word of column name) corresponding to that song is included, and if that word has been used many times the value will be large.
For example, "life ·" is used in many songs, so it becomes a small value.
On the other hand, "bicycl ·" is a big value because it is used only by two songs "Bicycle Race" and "More Of That Jazz".In addition, "Bicycle Race" is a song refraining "bicycle" many times, so the word "bicycle" has a larger value (46.557) than "More Of That Jazz" which appears only once.
Using a technique called SVD (Singular Value Decomposition) for this document word matrix, it is possible to reduce the information of the matrix to low dimensions, and by mapping the songs and words themselves using similar words, visual You can group them.
On the left side of the figure below, the similarity of music is visualized, and on the right side, word similarity is visualized.
On the left side of the figure above, songs nearby are similar, something far away are not similar.
Many points are located in the vicinity of the origin (0, 0), but labels are attached to the points (songs) further from those gatherings.The songs with these labels will be something with unique words in the entire songs of the Queen, but here we will focus on the two songs "Bohemian Rhapsody" and "Great King Rat" .
Both are located far from the origin, but looking at the similarity of the words from the right figure, "mama ·", "tell ·", "die ·" are located at the same distance from the two songs It is located * 1.
These words are somewhat peculiar words in the sense that they are not used in other songs because they are located somewhat far from the origin, but "Bohemian Rhapsody" and "Great King Rat" It is used to some extent.
In connection with these matters, two songs are located in such a place * 2.
By the way Freddie Mercury is writing songs of both.
* 1 The minimum value and the maximum value of each coordinate are different in the left figure and the right figure, but you can ignore it.
* 2 Actually, I am also considering the usage of other words.
As a related analysis, we will try (hierarchical) cluster analysis using singular value decomposed information (singular vector).In cluster analysis, songs using similar words (precisely the songs closest to the distance) are grouped one by one and tree diagrams can be created.
From this result, you can group songs by deciding the number of groups (clusters), but here we will show you the result of coloring for each group.
Mapping by singular value decomposition of document word matrix, cluster analysis
Can you distinguish lyrics from words contained in lyrics?
There were four members of Queen at the time (1973 - 1991) subject to this analysis, Freddie Mercury, Brian May, Roger Taylor, and John Deacon, but all of them were involved in lyrics and composition, Each hit song is sent out to the world.
As it is in texts, there are words that lyrics frequently use for lyrics, and others do not use them much, but words specific to that lyricist.
How about the case of the Queen.
Here we will examine whether songwriters can be distinguished by using the discriminant analysis focusing on songs that are only one specific singer, and using the document word matrix described above.
We use a document word matrix (weight TF IDF) with the top 100 words in use frequency to check whether the predicted lyrics match the actual lyrics.
Discrimination
Read the result displayed by clicking the link above, as follows.
Below the result, there is a table like the following, which is a crosstabulation table of the actual songwriter (Actual) and the lyricist (Predicted) predicted by discriminant analysis.
| Actual |
Predicted Count |
|---|
| Author |
Mercury |
May |
Taylor |
Deacon |
|---|
| Mercury |
42 |
1 |
0 |
0 |
|---|
| May |
0 |
36 |
0 |
0 |
|---|
| Taylor |
0 |
0 |
17 |
0 |
|---|
| Deacon |
1 |
0 |
0 |
11 |
|---|
The first line is Freddy Mercury, but when predicting lyrics by using the information of the top 100 words, there are 42 songs predicted predictably.On the other hand, although only one song actually was written by Freddie Mercury, he predicted it was Brian May, that is, it was predicted to be out.
"Percent Misclassified" displayed above the table is an index of misclassification rate, which is an incorrectly predicted proportion (calculated by mistakenly predicted number of songs / number of songs to be analyzed).
"Canonic Plot" is called a canonical plot and is a visual representation of the discrimination situation with two-dimensional coordinates.
Four people are positioned on the graph, but Roger Taylor is positioned far away from the other three.Therefore, it can be said that the tendency of words used for Roger Taylor's poem is quite different from the other three.
In fact, when we check the crosstab presented above, all the songs (17 songs) songwritten by Roger Taylor are predictive and the songs by three other lyrics are mistaken for songwriting by Roger Taylor I also understand that there is nothing.By the way, the point labeled on the graph is misidentified songs.
My analysis report including the analysis introduced this time is saved in the following place.
https://public.jmp.com/users/259
From now on, I will post interesting analysis results if I feel like it, so stay tuned.
