622199_zh-CN

liliconcom

一周中最不受欢迎的一天

星期六并不是一周中最不受欢迎的一天。

 

大约一年前，我开始从 Twitter 收集趋势词数据。 除了制作艺术视频之外，没有任何真正的目标。 除了几次断电之外，数据相当完整，每五分钟采样十个地方。

A Linux shell window logging the captures.A Linux shell window logging the captures.A Linux shell window logging the captures.A Linux shell window logging the captures.记录捕获的 Linux shell 窗口。

Twitter API 的典型结果如下所示：

["2022-10-06 21:20:00.366761", 
"#SheHulk", "Matt Ryan", "Colts", "Colts", "Mario", "Hines", 
"#TNFonPrime", "Bowser", "#ThursdayNightFootball", 
"Russell Wilson", "#FanDuelisTheGREATEST", "#INDvsDEN", 
"Judy Tenuta", "Quavo", "Graphic B", "Hunter Biden", "Saweetie", 
"Matty Ice", "Luigi", "Phillip Lindsay", "Caden Sterns", "Jovic", 
"Charles Martinet", "Marijuana", "Offset", "Kathy", "Dank Brandon", 
"Carina", "Pittman", "Sasse", "Brooklyn", "Commanders vs Bears", 
"Frank Reich", "Primetime", "Elvis", "Mushroom Kingdom", "Jamal Cain", 
"Toad", "Sara Lee", "Sam Ehlinger", "Darren Bailey", "Norville", 
"Deon Jackson", "Rodney Thomas", "Ryan Kelly", 
"Mumbles McTouchdown", "Melvin Gordon", "Ranboo", "Illumination", ""]

通常每个位置都会有 50 个热门术语。 美国的九个地点有很多共同点，而且趋势会持续五分钟。 即使是一个微小的趋势也会受到很多关注。 这是我用来加载文件并制作单词和频率表的 JSL。

// for this quick test I copied the json files to my desktop.
// probably should put them in their own directory...
dir = "$desktop/";
names = Filter Each( {j}, Files In Directory( dir ), Ends With( j, ".json" ) );
// I'll need a list of open data tables
dtfiles = {};
For Each( {file}, names, Insert Into( dtfiles, Open( dir || file, JSON Settings( Col( "/root", Type( "JSON Lines" ) ) ) ) ) );
// I'll use an associative array to gather the frequency info
WordToFrequency = [=> 0];
For Each( {dt}, dtfiles,
 For Each Row( dt,
  // there are a lot of ways to do this; here it makes a list of col2..col51 from the current row
  row = dt[Row(), 2 :: N Cols( dt )];
  // then process the list
  For Each( {word}, row, WordToFrequency[word] += 1 );
 )
);
// then make a data table from the associative array
dtWF = New Table( "wordfreq",
 addrows( N Items( WordToFrequency ) ),
 New Column( "word", character, values( WordToFrequency << getkeys ) ),
 New Column( "freq", numeric, values( WordToFrequency << getvalues ) )
);

该表按单词排序：

There are currently 127K words. The blank one is a filler when there are <50 trends.There are currently 127K words. The blank one is a filler when there are <50 trends.There are currently 127K words. The blank one is a filler when there are <50 trends.There are currently 127K words. The blank one is a filler when there are <50 trends.目前有 127K 字。 当趋势 <50 时，空白是填充项。

第一个有趣的问题是“最流行的词是什么？” 按频率排序：

Caturday seems to be about cats.Caturday seems to be about cats.Caturday seems to be about cats.Caturday seems to be about cats.星期六似乎是关于猫的。

就在那里。 星期六是一周中最不受欢迎的一天。

正如中提到的WebSocket ，我之前对这些数据做了一个无聊的可视化。 数据按城市（或美国或世界）排列，并具有时间戳。 我有一个单独的程序，可以从维基百科上抓取新闻项目，这可能会有所帮助。 也许不会;推特热门词汇似乎与实际新闻松散（充其量）相关。

（使用此脚本时请注意：它滥用了 html 和 xml 之间的相似性，将 html 转换为 JMP 的 xml 阅读器可以处理的内容。 在使用这个想法之前，您需要弄清楚为什么任何特定的 html 看起来不够像 xml 并修复它。 请参阅第 10 行周围的正则表达式——它删除了导致 xml 阅读器混乱的 html 位。 使用 python 和 beautiful soup 可能会更容易，但我还没有这样做。）

dtBig = New Table( "news",
 New Column( "date", Numeric, "Continuous", Format( "yyyy-mm-dd", 12 ), Input Format( "yyyy-mm-dd" ) ),
 New Column( "text", Character, "Unstructured Text" )
);

For( ddd = 1apr2023, ddd < Today(), ddd += In Days( 1 ),
 url = "https://en.wikipedia.org/wiki/Portal:Current_events/" || Format( ddd, "Format Pattern", "<YYYY>_<MONTH>_<D>" );
 txt = Load Text File( url );
 Length( txt );
 xml = Regex(
  txt,
  "||&nbsp;|(<script.>)|(<FORM.>)|(<OPTION.>)|crossorigin|data-mw-ve-target-container|<LINK.>|<meta.>|<INPUT.>|<IMG.>",
  "",
  GLOBALREPLACE
 );
 dt = Open(
  Char To Blob( xml ),
  XML Settings(
   Stack( 0 ),
   Row( "/html/body/div/div/div/main/div/div/div/div/div/div/ul" ),
   Row( "/html/body/div/div/div/main/div/div/div/div/div/div/ul/li/ul" ),
   Row( "/html/body/div/div/div/main/div/div/div/div/div/div/ul/li/ul/li/ul" ),
   Row( "/html/body/div/div/div/main/div/div/div/div/div/div/ul/li/ul/li/ul/li/ul" ),
   Row( "/html/body/div/div/div/main/div/div/div/div/div/div/ul/li/ul/li/ul/li/ul/li/ul" ),
   Col(
    "/html/body/div/div/div/main/div/div/div/div/div/div/ul/li",
    Column Name( "text" ),
    Fill( "Use Once" ),
    Type( "Original Text" ),
    Format( {"Best"} ),
    Modeling Type( "Continuous" )
   ),
   Col(
    "/html/body/div/div/div/main/div/div/div/div/div/div/ul/li/ul/li",
    Column Name( "text" ),
    Fill( "Use Once" ),
    Type( "Original Text" ),
    Format( {"Best"} ),
    Modeling Type( "Continuous" )
   ),
   Col(
    "/html/body/div/div/div/main/div/div/div/div/div/div/ul/li/ul/li/ul/li",
    Column Name( "text" ),
    Fill( "Use Once" ),
    Type( "Original Text" ),
    Format( {"Best"} ),
    Modeling Type( "Continuous" )
   ),
   Col(
    "/html/body/div/div/div/main/div/div/div/div/div/div/ul/li/ul/li/ul/li/ul/li",
    Column Name( "text" ),
    Fill( "Use Once" ),
    Type( "Original Text" ),
    Format( {"Best"} ),
    Modeling Type( "Continuous" )
   ),
   Col(
    "/html/body/div/div/div/main/div/div/div/div/div/div/ul/li/ul/li/ul/li/ul/li/ul/li",
    Column Name( "text" ),
    Fill( "Use Once" ),
    Type( "Original Text" ),
    Format( {"Best"} ),
    Modeling Type( "Continuous" )
   ),

  ),
  XML Wizard( 0 )
 );
 For Each Row( dt, dt:text = Regex( dt:text, "<[^>]+>", "", GLOBALREPLACE ) );
 dt << New Column( "date", Numeric, "Continuous", Format( "yyyy-mm-dd", 12 ), Input Format( "yyyy-mm-dd" ), <<SETEACHVALUE></SETEACHVALUE></IMG.></INPUT.></LINK.></OPTION.></FORM.></D></MONTH></YYYY>

This is recent at the time the picture was made. Sometimes these entries are updated/corrected/deleted later.This is recent at the time the picture was made. Sometimes these entries are updated/corrected/deleted later.This is recent at the time the picture was made. Sometimes these entries are updated/corrected/deleted later.This is recent at the time the picture was made. Sometimes these entries are updated/corrected/deleted later.这是拍摄照片时最近的照片。 有时这些条目稍后会更新/更正/删除。

 

好的，我研究了一下星期六多一点。 这可能与星期六有关。 我们来分析一下。 尝试收集所有以 ...day 结尾的流行词，并记住它发生的那一天...

days = [=> ];
For Each( {dt}, dtfiles,
 (dt:col) << datatype( "numeric" );
 For Each Row( dt, 
  row = Filter Each( {trend}, dt[Row(), 2 :: N Cols( dt )], (Ends With( Lowercase( trend ), "day" )) );
  dow = Day Of Week( dt:col );
  For Each( {trend}, row,
   If( !Contains( days, trend ),
    days[trend] = [0, 0, 0, 0, 0, 0, 0]
   );
   days[trend][dow] += 1;
  );
 );
);

哇。 有一堆（1568）。 看起来星期六和星期六有很强的联系。 也许星期四实际上是最不受欢迎的。

Sunday is 1, Monday is 2, ...Sunday is 1, Monday is 2, ...Sunday is 1, Monday is 2, ...Sunday is 1, Monday is 2, ...周日是 1 点，周一是 2 点，...

蔓延到周日可能是时区问题。

 

欢迎任何想法...感谢您的阅读。

首先看一下十一个月的热门词汇。 星期六并不是一周中最不受欢迎的一天。