470773_en-US

liliconcom

Re: JMP Workflow Challenge 1: Motif Extraction and Identification from Continuous Power Data

Glad to hear that the add-in works on Mac.

 

You're right, I forgot to add the filter for "sprint" back in after making the add-in as generic as possible. 

 

The following code:

• uses the add-in to get fresh data from Strava
• filters rows to include only "sprint" names
• filters columns to match what was posted in "2022_03_28 Strava Sprinting Training Data Set 5 Runs" data set
• identifies motifs
• adds script to data table to chart the results

 

names default to here(1);

// if add-in is found, get fresh data from Strava, otherwise just use current data table
if( file exists("$ADDIN_HOME(com.caes.ngambles.strava-get-activities)\main.jsl"),
 include("$ADDIN_HOME(com.caes.ngambles.strava-get-activities)\main.jsl");
 wait(0);
);

// get reference to current data table
dt = current data table();

// Clean up data set for specific motif extraction challenge
if( contains( dt << get column names(string), "name"),
 r = dt << get rows where(
  !contains(lower case(:name), "sprint")
 );
 try(dt << delete rows(r));
);
colList = {
 "name",
 "altitude",
 "distance",
 "grade_smooth",
 "lat",
 "lng"
};
for( k = 1, k <= n items(colList), k++,
 try(dt << delete column( colList(k) ));
);

// Find Regions that match target shapes
dt << New Column( "Shape Match",
 Numeric,
 nominal,
 Formula(
  thresh = 12500;
  shapeList = {
   {221, 245, 289, 327, 374, 431, 477, 518, 535, 518, 491, 451, 401, 350, 304, 238, 178, 161, 156, 106,  96,  96,  87,  85,  81,  79,  76,  74,  73, 72, 73, 72, 72, 72, 72, 72, 71, 70, 70, 70}, 
   {306, 321, 347, 384, 429, 476, 515, 546, 579, 582, 576, 542, 489, 442, 375, 348, 295, 275, 242, 185, 152, 121, 106,  98,  93,  89,  89,  85,  84, 83, 82, 82, 81, 80, 79, 79, 78, 78, 78, 78}, 
   {357, 390, 401, 428, 475, 505, 558, 604, 619, 634, 634, 625, 581, 530, 483, 444, 378, 357, 324, 282, 214, 186, 162, 153, 137, 132, 124, 109, 104, 97, 94, 90, 90, 89, 88, 87, 87, 85, 85, 86}  
  };
  minErr = Empty();
  For( k = 1, k <= N Items( shapeList ), k++,
   err = 0;
   For( n = 0, n < N Items( shapeList[k] ), n++,
    For( m = 1, m <= N Items( shapeList[k] ), m++,
     err += (:watts[(Row() + m) - n - 1] - shapeList[k][m]) ^ 2
    );
    err /= N Items( shapeList[k] ) - 1;
    minErr = Minimum( minErr, err );
   );
  );
  If( minErr <= thresh,
   flag = 1,
   flag = 0
  );
  flag;
 )
);

// Run Region
dt << New Column( "Run Region",
 Numeric,
 "Continuous",
 Formula(
  regionCount = Summation( k = 0, 5, Lag( :Shape Match, k ) );
  If( regionCount < 6 & region Count > 0 & :Shape Match == 1,
   nearStart = 1,
   nearStart = 0
  );
  If( nearStart == 0,
   ans = :Shape Match;
   ,
   ans = 0;
   If( :watts[row()] > 0 & :watts[row() +1] > 0,
    ans = :Shape Match;
   )
  );
  ans;
 )
);

// motif ID
dt << New Column( "Motif ID",
 Numeric,
 Nominal,
 Formula( 
  mID = 0;
  For( k = 1, k <= Row(), k++,
   If( :Run Region[k - 1] == 0 & :Run Region[k] == 1,
    mID++  
   )
  );
  If( :Run Region == 1, mID ); 
 )
);

// row state formula column
dt << New Column("Row State Formula", 
 Row State, 
 Row State, 
 Formula(
  If(
   Is Missing(:Motif ID), 
   Row State() = Combine States(Hidden State(1), Excluded State(1)),
   Row State()  
  )  
 )  
);

// linear interpolation of zero values within motif ID
dt << New Column( "watts modified",
 Numeric,
 Continuous,
 Formula(
  If( Is Missing( :Motif ID ),
   Empty(),
   If( :watts > 0,
    :watts,
    rb = Row();
    ra = Row();
    For( k = Row(), k >= 1, k--,
     If( :watts[k] != 0,
      rb = k;
      Break();
     )
    );
    For( k = Row(), k <= N Rows(), k++,
     If( :watts[k] != 0,
      ra = k;
      Break();
     )
    );
    deltaPower = :watts[rb] - :watts[ra];
    deltaTime = rb - ra;
    :watts[rb] + (deltaPower / deltaTime) * (Row() - rb); 
   )
  )
 )
);

// rank timestamp, Motif ID - x axis values to help graph each motif
dt << New Column( "X",
 Numeric,
 Continuous,
 Formula( Col Rank( :timestamp, :Motif ID ) )
);

// graph results script added to data table
dt << new script(
 "Graph Results",
 names default to here(1);
 dt = current data table();
 
 nw = new window(
  "Motif Extraction Results",
  h list box(
   gb1 = dt << Graph Builder(
    Show Control Panel( 0 ),
    Show Legend( 0 ),
    Show Footer( 0 ),
    Variables(
     X( :Name( "X" ) ),
     Y( :watts modified ),
     Wrap( :Motif ID )
    ),
    Elements( Line( X, Y, Legend( 6 ) ), Points( X, Y, Legend( 7 ) ) )
   )
   ,
   gb2 = dt << Graph Builder(
    Show Control Panel( 0 ),
    Show Footer( 0 ),
    Variables( X( :X ), Y( :watts modified ), Overlay( :Motif ID ) ),
    Elements(
     Smoother( X, Y, Legend( 5 ), Lambda( 0.0014 ) ),
     Points( X, Y, Legend( 6 ) )
    )
   )
  )
 );
);

// run script to make graphs
// eval( dt << get property("Graph Results") );

Re: JMP Workflow Challenge 1: Motif Extraction and Identification from Continuous Power Data

It does work on the Mac, however, if you enter the refresh token incorrectly the add-in will refuse to load and you have to uninstall and reinstall. Is this pulling all recent activities? Might be good to have search string to pull in just those with the title of interest (i.e. "sprint").

Re: JMP Workflow Challenge 1: Motif Extraction and Identification from Continuous Power Data

I am unable to test it on a Mac, so my default choice was Windows only.  Attached is an add-in that allows both Windows and Mac. I'm not aware of a reason it shouldn't work, but it's untested.

 

Please let me know if it runs successfully on a Mac.

Re: JMP Workflow Challenge 1: Motif Extraction and Identification from Continuous Power Data

Is there any reason this wouldn't work on the macOS? I just tried to load it and you may have created it for Windows only.

Re: JMP Workflow Challenge 1: Motif Extraction and Identification from Continuous Power Data

I have been thinking about my initial approach to this challenge and thought I could do it better so I have made a second attempt.

 

This time, I have defined several target shapes and then search through the data looking for regions where the shapes match reasonably well. My defined target shapes are:

 

This method requires fewer tuning parameters, and allows for more flexibility by allowing additional target shapes to be added easily. 

 

I have added a column (watts modified) that performs a linear extrapolation when the watts value drops out during a run.

 

I have also added a column (Row State Formula) that automatically hides and excludes data points that are not associated with a motif ID.

 

In my opinion, this is an improvement over my initial solution to the challenge.

 

This method identifies 41 runs from the "2022_03_28 Strava Sprinting Training Data Set 5 Runs.jmp" data set

 

 

The JSL code to process the Strava data is:

names default to here(1);
<BR />// uncomment the following two lines if you have installed the Strava - Get Activities add-in and want to automatically download the Strava data prior to identifying motifs<BR />// include("$ADDIN_HOME(com.caes.ngambles.strava-get-activities)\main.jsl");<BR />// wait(0);<BR />
dt = current data table();

// Find Regions that match target shapes
dt << New Column( "Shape Match",
 Numeric,
 nominal,
 Formula(
  thresh = 12500;
  shapeList = {{221, 245, 289, 327, 374, 431, 477, 518, 535, 518, 491, 451, 401, 350, 304, 238, 178, 161, 156, 106, 96, 96, 87, 85, 81, 79, 76,
  74, 73, 72, 73, 72, 72, 72, 72, 72, 71, 70, 70, 70}, {306, 321, 347, 384, 429, 476, 515, 546, 579, 582, 576, 542, 489, 442, 375, 348, 295,
  275, 242, 185, 152, 121, 106, 98, 93, 89, 89, 85, 84, 83, 82, 82, 81, 80, 79, 79, 78, 78, 78, 78}, {357, 390, 401, 428, 475, 505, 558, 604,
  619, 634, 634, 625, 581, 530, 483, 444, 378, 357, 324, 282, 214, 186, 162, 153, 137, 132, 124, 109, 104, 97, 94, 90, 90, 89, 88, 87, 87, 85,
  85, 86}};
  minErr = Empty();
  For( k = 1, k <= N Items( shapeList ), k++,
   err = 0;
   For( n = 0, n < N Items( shapeList[k] ), n++,
    For( m = 1, m <= N Items( shapeList[k] ), m++,
     err += (:watts[(Row() + m) - n - 1] - shapeList[k][m]) ^ 2
    );
    err /= N Items( shapeList[k] ) - 1;
    minErr = Minimum( minErr, err );
   );
  );
  If( minErr <= thresh,
   flag = 1,
   flag = 0
  );
 )
);

// Run Region - helps clean up regions found by shape match
dt << New Column( "Run Region",
 Numeric,
 "Continuous",
 Formula(
  regionCount = Summation( k = 0, 5, Lag( :Shape Match, k ) );
  If( regionCount < 6 & region Count > 0 & :Shape Match == 1,
   nearStart = 1,
   nearStart = 0
  );
  If( nearStart == 0,
   ans = :Shape Match;
   ,
   ans = 0;
   If( :watts[row()] > 0 & :watts[row() +1] > 0,
    ans = :Shape Match;
   )
  );
  ans;
 )
);

// motif ID
dt << New Column( "Motif ID",
 Numeric,
 Nominal,
 Formula( 
  mID = 0;
  For( k = 1, k <= Row(), k++,
   If( :Run Region[k - 1] == 0 & :Run Region[k] == 1,
    mID++  
   )
  );
  If( :Run Region == 1, mID ); 
 )
);

// row state formula column
dt << New Column("Row State Formula", 
 Row State, 
 Row State, 
 Formula(
  If(
   Is Missing(:Motif ID), 
   Row State() = Combine States(Hidden State(1), Excluded State(1)),
   Row State()  
  )  
 )  
);

// linear interpolation of zero values within motif ID
dt << New Column( "watts modified",
 Numeric,
 Continuous,
 Formula(
  If( Is Missing( :Motif ID ),
   Empty(),
   If( :watts > 0,
    :watts,
    rb = Row();
    ra = Row();
    For( k = Row(), k >= 1, k--,
     If( :watts[k] != 0,
      rb = k;
      Break();
     )
    );
    For( k = Row(), k <= N Rows(), k++,
     If( :watts[k] != 0,
      ra = k;
      Break();
     )
    );
    deltaPower = :watts[rb] - :watts[ra];
    deltaTime = rb - ra;
    :watts[rb] + (deltaPower / deltaTime) * (Row() - rb); 
   )
  )
 )
);

// rank timestamp, Motif ID - x axis values to help graph each motif
dt << New Column( "X",
 Numeric,
 Continuous,
 Formula( Col Rank( :timestamp, :Motif ID ) )
);

 

Re: JMP Workflow Challenge 1: Motif Extraction and Identification from Continuous Power Data

I'm glad to hear that the watts are coming in for you through the API.

 

I have made a few more updates to the Strava API code that handles automatically getting refreshed access codes, and then I rolled that up into a generic add-in that I've attached.

 

The first time you run the addin you will be prompted to enter your Strava API info

which gets saved in your add-in folder so you do not have to enter it again.

 

The add-in returns a data table containing all of the available activities from Strava.

 

This add-in should further streamline the process of getting data from the Strava API, not only for this project but for others who also want to explore their data from Strava.

 

Re: JMP Workflow Challenge 1: Motif Extraction and Identification from Continuous Power Data

Fantastic @ngambles . This works great and thanks for posting the link to the video as well, which describes how to get the access token. These data are coming in as I'd like to see them. And plus this is a nice improvement over manually exporting them and combining them to a .csv file. Really sets the stage for automating this data pull and getting the data cleaned up to be analyzed and published on a regular basis to JMP Live. And yes, the watts are coming in for me through the API from your script.

I pasted in @brady_brady 's formulas from above, and it still doesn't get all the intervals. However, thanks to  @ngambles 's Strava API code, we have another interesting feature in the data set to explore and use, which is the smoothed velocity column. I think this can be useful. You also now have the unique activity IDs to work with as another feature (knowing that I will only run between 6-10 sprints in any given activity. I think we should throw the ML model at it again, @Byron_JMP , now that we have a bigger training data set and a couple additional features to build the model.

I've attached the updated training data set.

And @Jordan_Hiller I think you should try too! You are a data cleanup pro.

Re: JMP Workflow Challenge 1: Motif Extraction and Identification from Continuous Power Data

I have taken a shot at getting data straight from the Strava API.  I am successfully getting measurements for distance, velocity, position, altitude, & grade. However I am not getting power measurements.  I have a suspicion that this is due to my Strava account level and/or due to my lack of power measurement equipment. I have reason to believe that my code might work for users that have power data in their Strava data, but I'm unable to confirm. If this solution isn't correct, I can't imagine that it is too far away from what is needed.

 

I found this youtube video very informative for getting started with the Strava API. It clearly demonstrates how to get your access token from the Strava API, which is required before the JSL code will get any information from Strava. In the code below, replace your_access_token_here with your access token you get from the Strava API prior to running. 

 

names default to here(1);

accessToken = "your_access_token_here";       // access token from the Strava API
                // helpful link to get started with the API: https://www.youtube.com/watch?v=sgscChKfGyg
//////////////////////////////////////////
//////////////////////////////////////////
// get list of activities from Strava
s = New HTTP Request(
 URL( "https://www.strava.com/api/v3/athlete/activities?access_token=" || accessToken ),
 Method( "GET" ),
 Headers( {"Accept: application/json"} )
) << Send;
dtx = json to data table(s);
dtx << set name("Activities List");
// keep only activites marked as sprint
r = dtx << get rows where(
 !contains(lowercase(:name), "sprint")
);
dtx << delete rows(r);
if( n rows(dtx) == 0,
 close(dtx, no save);
 throw();
);
// get list of unique activity ids
summarize( activityList = by(dtx:id));
// get and combine the data from each activity
dty = new table();
dty << set name("Strava Data_temp");
for( k = 1, k <= n items(activityList), k++,
 actStream = New HTTP Request(
  URL( "https://www.strava.com/api/v3/activities/" || activityList[k] || "/streams?keys=latlng,watts,distance,velocity,altitude,velocity_smooth,cadence,temp,moving,grade_smooth&access_token=" || accessToken ),
  Method( "GET" ),
  Headers( {"Accept: application/json"} )
 ) << Send;
 dtz = json to data table(actStream);
 dtz << new column("Activity ID", numeric, nominal, set each value(num(activityList[k])));
 dty << concatenate(dtz, append to first table);
 close(dtz, no save);
);
// add rank column for x value
dty << New Column( "x",
 Numeric,
 "Continuous",
 Formula( Col Rank( :Activity ID, :type, :Activity ID ) )
);
// split up latitude and longitude pairs
myCount = 0;
for( k = 1, k <= n rows(dty), k++,
 if( dty:type[k] != "latlng",
  dty:type[k] = dty:type[k]
  ,
  myCount ++;
  if( modulo(myCount,2) == 0,
   dty:type[k] = "lat",
   dty:type[k] = "lng"
  )
 );
);
// split tall table to wider table
dt = dty << Split(
 Split By( :type ),
 Split( :data ),
 Group( :x, :Activity ID ),
 Remaining Columns( Drop All ),
 Sort by Column Property
);
// done with this data table
close(dty, no save);
// update start time from activities table
dt << Update(
 With( dtx ),
 Match Columns( :Activity ID = :id ),
 Add Columns from Update Table( :start_date, :name ),
 Replace Columns in Main Table( :start_date, :name )
);
// close activity list
close(dtx, no save);
// clean up time stamp
dt << Begin Data Update;
dt << Recode Column(
 dt:start_date,
 {Regex( _rcNow, ".*^(.*)Z.*", "\1", GLOBALREPLACE )},
 Target Column( :start_date )
);
dt << End Data Update;
dt:start_date << data type(numeric) << Format( "yyyy-mm-ddThh:mm:ss", 19, 0 ) << Input Format( "yyyy-mm-ddThh:mm:ss", 0 );
dt << New Column( "timestamp",
 Numeric,
 "Continuous",
 Format( "yyyy-mm-ddThh:mm:ss", 19, 0 ),
 Input Format( "yyyy-mm-ddThh:mm:ss", 0 ),
 Formula( (:start_date + :x) - 1 )
);
wait(0);
dt:timestamp << delete formula;
dt << delete column("start_date");
// arrange columns
dt << Move Selected Columns( {:timestamp}, To first );
dt << Move Selected Columns( {:Activity ID, :name}, after(:timestamp) );
// set name
dt << set name("Strava Data");
// sort chronologically
dt << Sort( By( :timestamp ), Order( Ascending ), replace table );
dt << sort( by(:Activity ID), order( ascending ), replace table );

The code:

• looks through all Strava activities and selects only those with "sprint" in their names  (not case sensitive)
• tries to get many measurements including power (watts)
• organizes the information in a data table similar to the original data sets provided in this challenge

 

Re: JMP Workflow Challenge 1: Motif Extraction and Identification from Continuous Power Data

I took a shot a getting data straight from the Strava API. I am successfully getting several measurements including distance, velocity, longitude, latitude, altitude, and grade. However I am not getting power. I suspect that this may be due to my Strava account and/or my lack of equipment for measuring power. This solution may or may not be what is needed for someone with power information in their Strava account, but if it's not correct, I can't imagine that I'm too far off. 

 

To get started with the Strava API, this youtube video was very helpful and it clearly demonstrates the steps to get your access token, which is required before this code will get any data from Strava. In the code below, replace your_access_token_here with your actual access token, once you have it.

 

names default to here(1);

accessToken = "your_access_token_here";       // access token from the Strava API
                // helpful link to get started with the API: https://www.youtube.com/watch?v=sgscChKfGyg
//////////////////////////////////////////
//////////////////////////////////////////
// get list of activities from Strava
s = New HTTP Request(
 URL( "https://www.strava.com/api/v3/athlete/activities?access_token=" || accessToken ),
 Method( "GET" ),
 Headers( {"Accept: application/json"} )
) << Send;
dtx = json to data table(s);
dtx << set name("Activities List");
// keep only activities marked as sprint
r = dtx << get rows where(
 !contains(lowercase(:name), "sprint")
);
dtx << delete rows(r);
if( n rows(dtx) == 0,
 close(dtx, no save);
 throw();
);
// get list of unique activity ids
summarize( activityList = by(dtx:id));
// get and combine the data streams from each activity
dty = new table();
dty << set name("Strava Data_temp");
for( k = 1, k <= n items(activityList), k++,
 actStream = New HTTP Request(
  URL( "https://www.strava.com/api/v3/activities/" || activityList[k] || "/streams?keys=latlng,watts,distance,velocity,altitude,velocity_smooth,cadence,temp,moving,grade_smooth&access_token=" || accessToken ),
  Method( "GET" ),
  Headers( {"Accept: application/json"} )
 ) << Send;
 dtz = json to data table(actStream);
 dtz << new column("Activity ID", numeric, nominal, set each value(num(activityList[k])));
 dty << concatenate(dtz, append to first table);
 close(dtz, no save);
);
// add rank column for x value
dty << New Column( "x",
 Numeric,
 "Continuous",
 Formula( Col Rank( :Activity ID, :type, :Activity ID ) )
);
// split up latitude and longitude pairs
myCount = 0;
for( k = 1, k <= n rows(dty), k++,
 if( dty:type[k] != "latlng",
  dty:type[k] = dty:type[k]
  ,
  myCount ++;
  if( modulo(myCount,2) == 0,
   dty:type[k] = "lat",
   dty:type[k] = "lng"
  )
 );
);
// split tall table to wider table
dt = dty << Split(
 Split By( :type ),
 Split( :data ),
 Group( :x, :Activity ID ),
 Remaining Columns( Drop All ),
 Sort by Column Property
);
// done with this data table
close(dty, no save);
// update start time from activities table
dt << Update(
 With( dtx ),
 Match Columns( :Activity ID = :id ),
 Add Columns from Update Table( :start_date, :name ),
 Replace Columns in Main Table( :start_date, :name )
);
// close activity list
close(dtx, no save);
// clean up time stamp
dt << Begin Data Update;
dt << Recode Column(
 dt:start_date,
 {Regex( _rcNow, ".*^(.*)Z.*", "\1", GLOBALREPLACE )},
 Target Column( :start_date )
);
dt << End Data Update;
dt:start_date << data type(numeric) << Format( "yyyy-mm-ddThh:mm:ss", 19, 0 ) << Input Format( "yyyy-mm-ddThh:mm:ss", 0 );
dt << New Column( "timestamp",
 Numeric,
 "Continuous",
 Format( "yyyy-mm-ddThh:mm:ss", 19, 0 ),
 Input Format( "yyyy-mm-ddThh:mm:ss", 0 ),
 Formula( (:start_date + :x) - 1 )
);
wait(0);
dt:timestamp << delete formula;
dt << delete column("start_date");
// arrange columns
dt << Move Selected Columns( {:timestamp}, To first );
dt << Move Selected Columns( {:Activity ID, :name}, after(:timestamp) );
// set name
dt << set name("Strava Data");
// sort chronologically
dt << Sort( By( :timestamp ), Order( Ascending ), replace table );
dt << sort( by(:Activity ID), order( ascending ), replace table );

 

The code:

• only includes data for activities that contain "sprint", not case sensitive, in the name
• tries to get power (watts), among many other measurements including distance, velocity, position, altitude, etc...
• arranges the data in a format similar to the original data tables presented with this challenge

 

I do hope that this code works for a user that has power information in their Strava account.

Re: JMP Workflow Challenge 1: Motif Extraction and Identification from Continuous Power Data

Nice, Brady. 

 

Good point about some of the pretty short times from the start of an interval vs. a drop-out. The drop-outs do seem to be 2-3 seconds, max, however. We will see how this holds with new data being collected this week. 

 

Attached is some additional training data. This is about 1:40 of very steady state running, but at more of a rac -pace (nearly 100% of my critical power). It might help @Byron_JMP , though, with his ML model. Maybe add a feature to this training data set to bring it closer to the warm-up/warm-down range? 

 

Also who wants to tackle getting the data directly through the Strava API using HTTP Request (); in JMP? I know @dieter_pisot has done this before. I can tag the workouts with something like sprints so that the data pull can be easily retrieved through the API.

Re: JMP Workflow Challenge 1: Motif Extraction and Identification from Continuous Power Data

Here is another approach with interpolation of the dropouts, using some (relatively simple) formula columns, many of which could be eliminated by using a more complex interpolation formula, at the expense of easy troubleshooting and formula-writing. (See attached table for formulas and graph scripts)

 

The results seems consistent with what we want, at least in the sample dataset: (red points will be dropped)

 

 

Here is how it works so you can judge whether the assumptions made hold for your data:

 

First, interpolation of the zeros occurring due to "dropouts". Linear interpolation was chosen; splines and quadratics were considered but the results were inferior to my eye.

Note the "Cum Zeros Behind" through "Interpolated Power" columns in rows 4-6 below.

"Cum Zeros Behind" is an inclusive lookback for zeros.

"Cum Zeros Ahead" is an inclusive look-ahead for zeros.

"Zero Block" is the number of contiguous zeros in the block.

"Groups for Zeros" starts at 1 and increases whenever a new group of zeros occurs

Interpolation occurs when there is a short (fewer than 7) block of zeros, not starting on row 1.

 

****Note: In the sample data, the longest case of interpolation spanned 4 zeros, while the shortest "true" break between intervals was 7 zeros. This is a pretty slim margin for error, and is a possibly significant drawback of this approach. It is important that the shortest "true" break be longer than the longest "dropout".

 

As mentioned previously, you don't need most of these columns, although the formula for the interpolation will grow much uglier without them, and troubleshooting will be (much) more difficult.

 

After interpolation, only blocks of at least 7 consecutive zeros remain in the data (zeros possibly occurring in the first few rows excepted.)

 

The "Interval" column groups the data. It starts at 1 and increments whenever:

-the timestamp changes by more than 100, or

-interpolated power moves from 0 to some nonzero value

 

The "Post-spike" column is a flag that sets during power's ramp-down from the spike to steady(ish) state.

It is (re)set to zero:

- at row 1

- whenever the interval column changes

- whenever a new block of zeros occurs

 

It sets to 1 whenever interpolated power moves from some value above 90 to some power less than or equal to 90. Based on the data in the sample this seemed reasonable.

 

 

We can now decide which data to ignore--the purpose of the "Ignore" column. We ignore data when any of the following are true:

- The maximum over the interval is < 400. (This is the "background noise"). Again, based on the provided data.

- Interpolated power moves above 100, post-spike. (Indicating we've moved into "background noise", rather than ramping down towards 0)

- Interpolated power is 0

 

"Final Power" is the same as "Interpolated Power", but set to missing when "Ignore" == 1.

"Interval Time" computes time relative to the interval (i.e., starts at 0 and increments from there). Like "Final Power", it is set to missing whenever "Ignore" == 1.

 

I did not disable formula evaluations and subset out the non-ignored data, but this is a likely next step.

 

It will be interesting to see how Dan breaks this next

 

Cheers,

Brady

 

Re: JMP Workflow Challenge 1: Motif Extraction and Identification from Continuous Power Data

Here are all the solutions so far on JMP Public:

 

JMP Workflow Challenge 1: Motif Extraction and Identification from Continuous Power Data   View this at public.jmp.com

Re: JMP Workflow Challenge 1: Motif Extraction and Identification from Continuous Power Data

This is a really nice solution. I posted the output on JMP Public with a Local Data Filter to look at the new data set, the three training runs and your feature column of binary steady state:

Re: JMP Workflow Challenge 1: Motif Extraction and Identification from Continuous Power Data

@Daniel_Valente,

 

Here is my attempt at the challenge.

 

My approach is to identify features at the beginning and end of the motif, identify the entire run region (between the beginning and the end), then assign a motif ID to each.  

 

Here is my code for the column formulas that I used:

names default to here(1);

dt = current data table();

// date from time stamp
dt << New Column( "Date[timestamp]",
 Numeric,
 Nominal,
 Format( "yyyy-mm-dd", 19 ),
 Input Format( "yyyy-mm-dd" ),
 Formula( :timestamp - Time Of Day( :timestamp ) )
);

// rate of change
dt << New Column( "rate of change",
 Numeric,
 Continuous,
 Formula(
  Col Moving Average(
   Dif( 
    Col Moving Average(
     :power / Col Maximum( :power, :Name( "Date[timestamp]" ) ),
     0.6,
     5,
     5,
     Empty(),
     :Name( "Date[timestamp]" )
    )
   )  
   ,
   0.95,
   5,
   5,
   Empty(),
   :Name( "Date[timestamp]" )
  )
 )
);

// Start point
dt << New Column( "Start Point",
 Numeric,
 Continuous,
 Formula( 
  If( Row() == 1,
   0,
   checkZero = 1 * (:power[Row()] > 0 & :power[Row() - 1] == 0);
   foundPeak = 0;
   For( k = Row() + 1, k <= N Rows(), k++,
    If( :power[k] > Col Quantile( :power, 0.93, :Name( "Date[timestamp]" ) ),
     foundPeak = 1;
     Break();
    );
    If( :power[k] == 0, Break() );
   );
   If( checkZero == 1 & foundPeak == 1,
    1,
    0
   );
  )
 )
);

// steady state
dt << New Column( "Steady State",
 Numeric,
 Continuous,
 Formula(
  ans = 1;
  For( k = 0, k <= 10, k++,
   ans = Minimum(
    ans,
    1 * (:rate of change[Row() - k] < 0.003 &
    :rate of change[Row() - k] > -0.003)
   )
  );
  ans;
 )
);

// Run Region
dt << New Column( "Run Region",
 Numeric,
 Continuous,
 Format( "Best", 12 ),
 Formula(
  ans = 0;
  For( k = Row(), k >= 1, k--,
   If( :Start Point[k] == 1,
    ans = 1;
    Break();
   );
   If( :Steady State[k] == 0 & :Steady State[k - 1] == 1,
    ans = 0;
    Break();
   );
  );
  ans;
 )
);

// motif ID
dt << New Column( "Motif ID",
 Numeric,
 Nominal,
 Formula( 
  If( 
   :Run Region == 1,
   Col Cumulative Sum( :Start Point )   
  )  
 )
);

// rank timestamp, Motif ID - helps plot each motif
dt << New Column( "Rank[timestamp]",
 Numeric,
 Continuous,
 Formula( Col Rank( :timestamp, :Motif ID ) )
);

Re: JMP Workflow Challenge 1: Motif Extraction and Identification from Continuous Power Data

Nice, Byron -- this looks pretty good. It's picking out those two noise curves. I'll collect some additional data over the weekend and see how it fairs.

Re: JMP Workflow Challenge 1: Motif Extraction and Identification from Continuous Power Data

Nice! That sounds like a good solution too. Post the JSL if you can dig it up and we'll try it.

Re: JMP Workflow Challenge 1: Motif Extraction and Identification from Continuous Power Data

Hmm. Good questions. Always good to explore assumptions when thinking about a data problem. And also ask the people who own the system that is generating the data!

 

1) Only one cooldown warmup per session. Warmup doesn't really vary that much. Mean is like ~250 +-30 watts unless I've had a few cups of coffee... 

2) As you can see I really can only hold my top speed for 6-10 seconds and then the power starts dropping. Maybe in the future they will be longer/wider. We'll have to explore that.

3) Zeros after the walking period shouldn't be included. They are a reset period. It's largely 0 watts prep from start. Sprint for a period of time. Slow down to a stop. Walk back. Return to zero for the set of the next interval.

4) Zeros during the motif are bad data and should be interpolated. I've not stopped. It's just that the sensor has dropped out. They should be fixed, marked as missing or interpolated through otherwise I could see it causing some issues with the subsequent analysis.

 

Re: JMP Workflow Challenge 1: Motif Extraction and Identification from Continuous Power Data

I changed the simple column formula approach to include an AI model, so now the table has one fantastically complex formula along with some simpler ones.

The attached script add columns and a graph. The model scores the curve segments as signal or noise. 

I'm interested to see how it will score a unique data set. 

 

Re: JMP Workflow Challenge 1: Motif Extraction and Identification from Continuous Power Data

I have done something a bit similar at work where I was counting temperature cycles for thermal shock oven. I think in that case I did it with couple of different formula columns, I'll have to take a look at the code at work and see if it could be modified for this case.

 

If I remember correctly my idea was to detect "start point" of the cycle (0 power in this case and get last value of those), then check that there are enough values which are over some threshold (here it could be 5 values over 350) and after that check where cycle ends (flat part with ~20 values between 80-100). And then have some running variables (motid_id, cycle_ongoing, ...) which I would increase and reset as needed.

Re: JMP Workflow Challenge 1: Motif Extraction and Identification from Continuous Power Data

Assumptions matter. What are you willing to assume about the data? There's where your issues will lie.

 

1) how many cooldown/warmups are possible? Do you know the mean power for these? Does it vary?

2) Motifs can vary in length... will they, however, always be of similar length, whatever that is? In the sample data thus far, this has been the case... will it remain the case going forward?

3) What about zeros? Should they be included as the leading part of the motif, or not? All of them, a certain number of them, none of them?

4) What about zeros that occur mid-motif? Are these legitimate? What about non-zero points mid-motif? How can we tell whether these are legitmate? Should we try?

 

etc.etc.etc

 

Cheers,

Brady

 

Re: JMP Workflow Challenge 1: Motif Extraction and Identification from Continuous Power Data

Here it is with the third file concatenated with @Byron_JMP 's script. Looks like the same thing occurs. We're tagging part of the area to exclude as a motif. Good thing is that it seems to fail in the same spot or using the same failure mode. You can see it in the cycle group 9 and 18 and also in the sample/power plot.

Re: JMP Workflow Challenge 1: Motif Extraction and Identification from Continuous Power Data

Ooo! Nice @Byron_JMP . I like the formula column solution. Just tried to scale it. It fails in a different way than @brady_brady 's solution.

 

Motif 9 looks a little suspect...

Re: JMP Workflow Challenge 1: Motif Extraction and Identification from Continuous Power Data

Uh oh! Problem when I am trying to scale this. We now have two days of data. Ran Brady's script to identify hopefully 16 Motifs.

 

I have new data coming in now and ran the script @brady_brady wrote. Correctly identified 16 motifs. But there is some oddity in the 8th one now. Looks like the algorithm is confused by the new data. I do have the time stamp, maybe that could help in making this scale?

 

Off to try @Byron_JMP 's solution.

Re: JMP Workflow Challenge 1: Motif Extraction and Identification from Continuous Power Data

@brady_brady That's a pretty slick method for solving the problem.

I did it with three simple column formulas.

 

 

new column("time point", formula(If( Row() == 1,
 1,
 If( :power == 0 & Lag( :time point, 1 ) <= 50,
  Lag( :time point, 1 ) + 1,
  If( :power == 0,
   1,
   Lag( :time point, 1 ) + 1
  )
 )
)));

new column("segments", formula(
If(
 Row() == 1, 0,
 :time point == 1, Lag( :segments, 1 ) + 1,
 Lag( :segments, 1 )
)));


new column("cycle", formula(
If( :time point > 70 | :segments == 0,
 .,
 :segments
)));

Re: JMP Workflow Challenge 1: Motif Extraction and Identification from Continuous Power DataNice, Brady! Great work. It’s interesting that there is some data quality problems, too, I can fit a peak model to smooth that out. Maybe those could be marked as missing because they are not really 0, or interpolated?Re: JMP Workflow Challenge 1: Motif Extraction and Identification from Continuous Power Data

It is surprising how far you can get with a single formula column:

 

dt << new column("Motif ID", nominal, formula(Col Cumulative Sum(
 Row() > 6 & Summation( i = 1, 6, Abs( Lag( :power, i ) ) ) == 0 & :power != 0
)));

However, there is still some cleanup to do... this approach creates a "0" group ID, which is the leading data, and does not cut off the last motif where we'd like. The "6" in the summation is because I designate a new group if at least 6 consecutive zeros are encountered prior to a power change.

 

There are of course many other ways to do this that are more sophisticated, involving comparisons to the background floors, attacks/decays, etc... but if a simple approach like this one works, great.

 

The script below incorporates this idea, but cleans up the dataset, as well. Ideally the values in the last few lines would be parameterized.

 

names default to here(1);

dt = current data table();

dtSub = dt << subset(selected rows(0), selected columns(0));

//create ID column
dtSub << new column("Motif ID", nominal, formula(Col Cumulative Sum(
 Row() > 6 & Summation( i = 1, 6, Abs( Lag( :power, i ) ) ) == 0 & :power != 0
)));
dtSub:MotifID << delete formula;

//the next 2 loops truncate the dataset, assuming it ends like the sample dataset ended.
for(i  = nrow(dtSub), abs(dtSub:power[i] - dtSub:power[i-1]) < 20, i--,
 dtSub << Delete Rows(i)
);

for(i  = nrow(dtSub), abs(dtSub:power[i] - dtSub:power[i-1]) > 5, i--,
 dtSub << Delete Rows(i)
);

//delete ID 0 group
dtSub << select where(:Motif ID == 0) << delete rows;

//delete 0 power rows, but not when they lie within a group
i = nrow(dtSub)-1;
while (i > 1,
 if(dtSub:Motif ID[i] < dtSub:MotifID[i+1] & dtSub:power[i] == 0,
  while( dtSub:power[i] == 0,
   dtSub << delete rows(i--);
  )
 );
 i--
);
 

 

 

 

JMP Workflow Challenge 1: Motif Extraction and Identification from Continuous Power Data

Calling JMP workflow and JSL wizards. Here's a Thursday challenge for you. The attached data set is set of power data off my running power meter. What we want to do is identify and correctly tag the 8 motifs that occur during this ~1000 point signal data set attached to this post, create a new column with function ID and write it in a way that it can be used to automate the tagging of future data that follows this same envelope. 

 

The data collection assumptions for this data set and the future are:

• Prior to the start of a motif the power is 0.
• There is a period before and after the motifs that will be non-zero (warm-up and warm-down). This is not part of the system under test and should not be tagged.
• Each motif has an attack, sustain, decay and release. The shape of the function is identical from motif to motif. It goes from 0 to some peak over a period of samples and then ramps down to the baseline noise floor (walking power). The motif ID is over when the power reads 0 again.
• There will be an unknown number of functions to identify in the future. There are 8 in the data set. The solution needs to be invariant to that.
• The overall length of the functions as well as the peak can vary.

We are working on the left edge of the analytic workflow today (shown in green below). The data comes from a file (.csv), and eventually when we automated this workflow a folder of n .csv files. The Data Access is being used, and what the task is today is to perform the Data Blending and Cleaning tasks on this example file to have the data ready to expand the workflow to other analytic capabilities in the future.

Leave your solutions in the comments.

 

 

 

Data Blending and Cleanup