A. L. Seidel: Tracking Project

I have been working on a program to track objects to be used in the introductory physics classes here at SCSU. It is being done in Visual C++ 2008 due largely to the SDK provided by The Imaging Source being in Visual Studio, though I could have used another C++ compiler I would have been starting even more from scratch.
The program set up allows the user to adjust the sensitivity levels for the three colors. This works as a multiplier requirement, such as by default for a pixel to be considered red it must have a red level that is 1.8 times both the green and the blue levels. The user can also select which colors they want to track choosing 1 to 3 of the colors. Finally the user calibrates the distance by adjusting the spacing between the white lines until the ends of a meter stick match two of them. Then they count the number of lines that the meter stick is long and adjust that value. I have been having issues with my install of Visual Studio so until I can get it to find and load the System.Windows.Forms.dll the output file is a static name, ideally I want the student to be able to choose the location and name of their output files.
As of right now I am able to get 16.4/13.5/12.8 data points per second when tracking one/two/three objects. This needs to be higher to be accurate enough for the students to be able to achieve the right results. In the last test I preformed before writing this post I was able to get a measurement of the acceleration due to gravity on earth of 9.71 meters per second squared (should be 9.81 m/s^2 a 1% relative error) . This is off a little possibly due to not exactly calibrating, the balls being stress balls encounter air resistance, and the software not being as fast as it should. Below is the imaging processing part of the code which also does the file output. If anyone looks at this and has ideas to increase the efficiency of the code please let me know.

**Update - Below is the current version of the code which has had some suggestions added. Preliminary test show 25+ data points per second tracking one object, so an improvement. Thank you to those of you who have offered suggestions so far.

***Update - Adjusted file output. With VLC playing a video and a bunch of other apps running I am getting 41/25 points per second when tracking 1/3 objects. Thanks for all the suggestions, anymore suggestions are still welcome. Thank you again.


.

.

.



FILE *fp;

SYSTEMTIME systime, FileTime;

int StartTime = 0, CurrentTime = 0, ListItems=0;

char FileNameString[100];



struct RGB24Pixel

{

  BYTE b;

  BYTE g;

  BYTE r;

};





typedef struct _LIST_DATA

{

	SLIST_ENTRY ItemEntry;

	int msTime;	// time in miliseconds

	int rx;		// x position of the center of red

	int ry;		// y position of the center of red

	int bx;		// x position of the center of blue

	int by;		// y position of the center of blue

	int gx;		// x position of the center of green

	int gy;		// y position of the center of green

}LIST_DATA, *PLIST_DATA;



.

.

.



//////////////////////////////////////////////////////////////////////////

// The image processing is done here. 

//////////////////////////////////////////////////////////////////////////



void

CListener::DoImageProcessing (smart_ptr pBuffer)

{

  // Get the bitmap info header from the membuffer. It contains the bits per pixel,

  // width and height.

  smart_ptr pInf = pBuffer->getBitmapInfoHeader ();



  // Now retrieve a pointer to the image. For organization of the image data, please

  // refer to:

  // http://www.imagingcontrol.com/ic/docs/html/class/Pixelformat.htm



  extern int RedThresholdLevel;

  extern int BlueThresholdLevel;

  extern int GreenThresholdLevel;

  extern int Lines;

  extern int LineSpacing;

  extern bool TrackRed;

  extern bool TrackBlue;

  extern bool TrackGreen;

  extern bool Tracking;



  int Calibration = Lines*LineSpacing;

  int RedCount = 0;

  int BlueCount = 0;

  int GreenCount = 0;



  int RedCenterSumX = 0;

  int RedCenterSumY = 0;

  int BlueCenterSumX = 0;

  int BlueCenterSumY = 0;

  int GreenCenterSumX = 0;

  int GreenCenterSumY = 0;



  int RedCenterX = 0;

  int RedCenterY = 0;

  int BlueCenterX = 0;

  int BlueCenterY = 0;

  int GreenCenterX = 0;

  int GreenCenterY = 0;



  float RedThresholdMultiplier = (((float) RedThresholdLevel) / (float) 100);

  float BlueThresholdMultiplier = (((float) BlueThresholdLevel) / (float) 100);

  float GreenThresholdMultiplier = (((float) GreenThresholdLevel) / (float) 100);







  RGB24Pixel* pbImgData = (RGB24Pixel*) pBuffer->getPtr ();

  SIZE dim = pBuffer->getFrameType ().dim;

  //int iOffsUpperLeft = (dim.cy-1) * dim.cx;



  //BYTE* pImageData = pBuffer->getPtr();



  // Calculate the size of the image.

  //int iImageSize = pInf->biWidth * pInf->biHeight * pInf->biBitCount / 24 ;



  // Now loop through the data and change every byte.

  //for( int i = 0; i < iImageSize; i++)

  //{

  int i = 0;

  //CTime cTime = CTime::GetCurrentTime();

  //printf("%s",cTime);

  for (int ix = 0; ix < dim.cx; ix++)

    {

      for (int iy = 0; iy < dim.cy; iy++)

        {

          i = ix + dim.cx*iy;



		  if ((TrackRed) && (pbImgData[i].r > RedThresholdMultiplier * pbImgData[i].b) && (pbImgData[i].r > RedThresholdMultiplier * pbImgData[i].g))

            {

              pbImgData[i].b = 0;    // BLUE

              pbImgData[i].g= 0;    // GREEN

              pbImgData[i].r = 255;    // RED



              RedCount++;

              RedCenterSumX += ix;

              RedCenterSumY += iy;



            }

          else if ((TrackBlue) && (pbImgData[i].b > BlueThresholdMultiplier * pbImgData[i].r) && (pbImgData[i].b > BlueThresholdMultiplier * pbImgData[i].g))

            {

              pbImgData[i].b = 255;    // BLUE

              pbImgData[i].g= 0;    // GREEN

              pbImgData[i].r = 0;    // RED



              BlueCount++;

              BlueCenterSumX += ix;

              BlueCenterSumY += iy;



            }

          else if ((TrackGreen) && (pbImgData[i].g > GreenThresholdMultiplier * pbImgData[i].r) && (pbImgData[i].g > GreenThresholdMultiplier * pbImgData[i].b))

            {

              pbImgData[i].b = 0;    // BLUE

              pbImgData[i].g= 255;    // GREEN

              pbImgData[i].r = 0;    // RED



              GreenCount++;

              GreenCenterSumX += ix;

              GreenCenterSumY += iy;



            }



			if ( !(ix % LineSpacing) || !(iy % LineSpacing))

            {

              pbImgData[i].b = 255; // BLUE

              pbImgData[i].g = 255; // GREEN

              pbImgData[i].r = 255; // RED

            }

        }

    }

  //}



  //Center of mass calculations

  RedCenterX = (int) ((float) RedCenterSumX / (float) RedCount);

  BlueCenterX = (int) ((float) BlueCenterSumX / (float) BlueCount);

  GreenCenterX = (int) ((float) GreenCenterSumX / (float) GreenCount);



  RedCenterY = (int) ((float) RedCenterSumY / (float) RedCount);

  BlueCenterY = (int) ((float) BlueCenterSumY / (float) BlueCount);

  GreenCenterY = (int) ((float) GreenCenterSumY / (float) GreenCount);



  //verify the positions are within valid range

  if (RedCenterX < 0)

    {

      RedCenterX = 0;

    }

  if (RedCenterY < 0)

    {

      RedCenterY = 0;

    }

  if (RedCenterX > dim.cx)

    {

      RedCenterX = dim.cx - 1;

    }

  if (RedCenterY > dim.cy)

    {

      RedCenterY = dim.cy - 1;

    }



  if (BlueCenterX < 0)

    {

      BlueCenterX = 0;

    }

  if (BlueCenterY < 0)

    {

      BlueCenterY = 0;

    }

  if (BlueCenterX > dim.cx)

    {

      BlueCenterX = dim.cx - 1;

    }

  if (BlueCenterY > dim.cy)

    {

      BlueCenterY = dim.cy - 1;

    }



  if (GreenCenterX < 0)

    {

      GreenCenterX = 0;

    }

  if (GreenCenterY < 0)

    {

      GreenCenterY = 0;

    }

  if (GreenCenterX > dim.cx)

    {

      GreenCenterX = dim.cx - 1;

    }

  if (GreenCenterY > dim.cy)

    {

      GreenCenterY = dim.cy - 1;

    }





  //Draw Center of mass location as a 10 by 10 Box of lighter color

  //Changed to Crosshair lines to prevent error

  if (TrackRed)

    {

      for (int ix = 0; ix < dim.cx; ix++)

        {

          i = ix + dim.cx*RedCenterY;

          pbImgData[i].b = 50; // BLUE

          pbImgData[i].g = 50; // GREEN

          pbImgData[i].r = 255; // RED

        }

      for (int iy = 0; iy < dim.cy; iy++)

        {

          i = RedCenterX + dim.cx*iy;

          pbImgData[i].b = 50; // BLUE

          pbImgData[i].g = 50; // GREEN

          pbImgData[i].r = 255; // RED

        }

    }



  if (TrackBlue)

    {

      for (int ix = 0; ix < dim.cx; ix++)

        {

          i = ix + dim.cx*BlueCenterY;

          pbImgData[i].b = 255; // BLUE

          pbImgData[i].g = 50; // GREEN

          pbImgData[i].r = 50; // RED

        }

      for (int iy = 0; iy < dim.cy; iy++)

        {

          i = BlueCenterX + dim.cx*iy;

          pbImgData[i].b = 255; // BLUE

          pbImgData[i].g = 50; // GREEN

          pbImgData[i].r = 50; // RED

        }

    }





  if (TrackGreen)

    {

      for (int ix = 0; ix < dim.cx; ix++)

        {

          i = ix + dim.cx*GreenCenterY;

          pbImgData[i].b = 50; // BLUE

          pbImgData[i].g = 255; // GREEN

          pbImgData[i].r = 50; // RED

        }

      for (int iy = 0; iy < dim.cy; iy++)

        {

          i = GreenCenterX + dim.cx*iy;

          pbImgData[i].b = 50; // BLUE

          pbImgData[i].g = 255; // GREEN

          pbImgData[i].r = 50; // RED

        }

    }









  if (Tracking)

    {

      GetSystemTime (&systime);

      if (StartTime == 0)

        {

			 // Initialize the list header.

			InitializeSListHead(&ListHead);

			 StartTime = ((60 * systime.wMinute + systime.wSecond)*1000 + systime.wMilliseconds);

			GetSystemTime (&FileTime);





			time_t rawtime;

			struct tm * timeinfo;

			time ( &rawtime );

			timeinfo = localtime ( &rawtime );



			// Create file name string, with date and time

			strftime(FileNameString, 100, "C:/Users/Administrator/Desktop/Output/SCSUPhysics %Y-%m-%d %H%M.csv", timeinfo);





			//Add Collumn titles and create file

			fp = fopen (FileNameString, "w");

			fprintf (fp, "Time(ms)");

			if (TrackRed)

			  {

				  fprintf (fp, ", ,Red x(m),Red y(m)");

				}

			  if (TrackBlue)

				{

				  fprintf (fp, ", ,Blue x(m),Blue y(m)");

				}

			  if (TrackGreen)

				{

				  fprintf (fp, ", ,Green x(m),Green y(m)");

				}

			  fprintf (fp, "\n");



			  fclose (fp);

        }



      CurrentTime = ((60 * systime.wMinute + systime.wSecond)*1000 + systime.wMilliseconds);



		pListData = new LIST_DATA;

		pListData->msTime = CurrentTime;

		pListData->rx = RedCenterX;

		pListData->ry = RedCenterY;

		pListData->bx = BlueCenterX;

		pListData->by = BlueCenterY;

		pListData->gx = GreenCenterX;

		pListData->gy = GreenCenterY;

		FirstEntry = InterlockedPushEntrySList(&ListHead,&pListData->ItemEntry);

		ListItems++;



	  if(ListItems>300)

	  {

		 fp = fopen (FileNameString, "a");



		 for( int Count = 0; Count < ListItems; ++Count )

		{

			ListEntry = InterlockedPopEntrySList(&ListHead);

			pListData = (PLIST_DATA)( ListEntry );

			fprintf (fp, "%d", pListData->msTime - StartTime);

			if (TrackRed)

			{

			  fprintf (fp, ", ,%g,%g", (float) pListData->rx / (float) Calibration, (float)  pListData->ry / (float) Calibration);

			}

			if (TrackBlue)

			{

			  fprintf (fp, ", ,%g,%g", (float) pListData->bx / (float) Calibration, (float)  pListData->by / (float) Calibration);

			}

			if (TrackGreen)

			{

			  fprintf (fp, ", ,%g,%g", (float) pListData->gx / (float) Calibration, (float)  pListData->gy / (float) Calibration);

			}

			fprintf (fp, "\n");

	      



			// free( pListData );

			delete pListData;

		}

		 fclose (fp);

		 ListItems=0;



	  }







    }



  if(!Tracking && (ListItems!=0))

  {

	 fp = fopen (FileNameString, "a");



	 for( int Count = 0; Count < ListItems; ++Count )

    {

        ListEntry = InterlockedPopEntrySList(&ListHead);

        pListData = (PLIST_DATA)( ListEntry );

		fprintf (fp, "%d", pListData->msTime - StartTime);

		if (TrackRed)

        {

          fprintf (fp, ", ,%g,%g", (float) pListData->rx / (float) Calibration, (float)  pListData->ry / (float) Calibration);

        }

		if (TrackBlue)

        {

          fprintf (fp, ", ,%g,%g", (float) pListData->bx / (float) Calibration, (float)  pListData->by / (float) Calibration);

        }

		if (TrackGreen)

        {

          fprintf (fp, ", ,%g,%g", (float) pListData->gx / (float) Calibration, (float)  pListData->gy / (float) Calibration);

        }

		fprintf (fp, "\n");

      



        // free( pListData );

        delete pListData;

    }

	 fclose (fp);

	 ListItems=0;



  }





}

2 comments:

Unknown said...: There's a few things that I see

1) There's a lot of math that is performed over and over. You should perform this once and reuse the saved value for the checks.

E.G.
if((TrackRed)&&(pbImgData[i].r>(((float)RedThresholdLevel)/(float)100)*pbImgData[i].b)&&(pbImgData[i].r>(((float)RedThresholdLevel)/(float)100)*pbImgData[i].g))

I'd pull out (((float)RedThresholdLevel)/(float)100)
save that off, and just reference each time you need to compute the ratios. If you pull this out (and similar green/blue) of the double for loops, that will significantly speed up the function as you save 6 divides per pixel.

2) optimize the if ladder. If you only care if you have (TrackRed) set or not, perform that first, then if the math functions inside the bracket, perform the computation necessary. This setup saves performing more math for each if statement specially if you don't care about it.

E.G.
float RedThresholdLevelRatio=(((float)RedThresholdLevel)/(float)100)

if(TrackRed)
{
if ((pbImgData[i].r>RedThresholdLevelRatio*pbImgData[i].b)&&(pbImgData[i].r>RedThresholdLevelRatio*pbImgData[i].g))
{

3)after processing your image to see the center of the R G and B, you then redraw a square of the object. To do this you go through the whole picture again and then compute if you need to change the pixel color. This is a big no no. Change it so you only access for loops in a tight pattern for pixels you need to change.

To fix this, change it to a if ladder:

if (TrackRed)
{
for(int iy=RedCenterY-5;iy < RedCenterY+5; iy++)
{
for(int ix=RedCenterX-5;ix < RedCenterX+5; ix++)
{
pbImgData[i].b = 100; // BLUE
pbImgData[i].g= 100; // GREEN
pbImgData[i].r = 255; // RED
}
}
}

4) Finally, separate your file access FULLY from your function. With it included, it takes significant delays in code to wait for the operating system to open a file, write to the file, and then close the reference. Easily on the order of 20 ms to 100 ms each time.
By separating this, and creating a separate thread that will perform the actions and allow the main thread to perform the computations only. You will want to create the thread outside the function, so it allows both to run all the time. (if you do it inside the function, then you'd be creating and destroying threads which won't save anything for time) and then pass the data into a queue for it to process (could be as simple as three FIFO queues. Place all necessary math you can in the file i/o thread since it won't really be doing much anyways.

Hope this helps.; July 21, 2009 at 9:01 PM
Unknown said...: if you have bool Tracking = false; what is the timing for tracking 1 object? I'm thinking file access is taking a lot of time.

It takes about 5.5 ms for the arm to move over to the data, then you have rotational latency. (Granted this is minimal). Under best conditions, this is about 6 ms per read/write. (You're opening and closing the file twice, so a total of 4 access to the file)

If you manage 25 samples per second, then you're averaging 40 ms per image.

Second, you missed the laddered if statements, and I see you have a complicated way of drawing the linespacings. Try this:

for (int ix = 0; ix < dim.cx; ix++)
{
for (int iy = 0; iy < dim.cy; iy++)
{
i = ix + dim.cx*iy;
if ((TrackRed)
if ((pbImgData[i].r > RedThresholdMultiplier * pbImgData[i].b) && (pbImgData[i].r > RedThresholdMultiplier * pbImgData[i].g))
{
RedCount++;
RedCenterSumX += ix;
RedCenterSumY += iy;
}
if ((TrackBlue)
if (pbImgData[i].b > BlueThresholdMultiplier * pbImgData[i].r) && (pbImgData[i].b > BlueThresholdMultiplier * pbImgData[i].g))
{
BlueCount++;
BlueCenterSumX += ix;
BlueCenterSumY += iy;
}
if ((TrackGreen)
if (pbImgData[i].g > GreenThresholdMultiplier * pbImgData[i].r) && (pbImgData[i].g > GreenThresholdMultiplier * pbImgData[i].b))
{
GreenCount++;
GreenCenterSumX += ix;
GreenCenterSumY += iy;
}
if ( !(ix % LineSpacing) || !(iy % LineSpacing))
{
pbImgData[i].b = 255; // BLUE
pbImgData[i].g = 255; // GREEN
pbImgData[i].r = 255; // RED
}
}
}; July 23, 2009 at 8:30 PM

A. L. Seidel

Tuesday, July 21, 2009

Tracking Project

2 comments:

Blog Archive

Twitter Updates