Unit 24 Task 3

D1/P6 – Design a control system that uses different types of sensors 

For my project I decided to use the Intellibot software to make a TypeRacer bot. 

My initial idea was to create a bot that would play CookieClicker for you and it would buy the upgrades so it wouldn’t be too boring however when I setup the first part of clicking the cookie it was having issues recognizing it and it was very slow and clicking it. It only clicked the cookie once every seven seconds.  

After this idea failing, I decided to make a TypeRacer bot that I felt more confident would succeed. My initial idea with the TypeRacer bot was for it to click the ‘Enter Typing Race’ button so it would join a race and then once the race starts it can type out the text at a speed the user chose beforehand. I also planned on trying to make it solve the captcha as well, but I decided I’ll cross that bridge when I get to it, so it was best to just get the main basic part of the program completed and working. 

This would make my bot do two things to get the first Distinction. First it would type out the text and, in most cases, win the race. The second thing it would do is to solve the captcha. 

The typing out text was fairly easy to setup, the captcha I wasn’t too sure I decided to deal with it later.  

First I decided to make a loop that would ask the user to enter a typing speed between 0.02 and 0.09 as you can see below this basically in programming would just be a while loop where the condition would become false when you enter a typing speed in that range. I also had to add an additional check that the user did not enter 0 because there was some issue where Intellibot would allow 0 as valid input that would pass the range check between 0.02 and 0.09. If the user successfully typed a number in this range it would then open up Google Chrome and browse to the website.  

Once it had opened up Google Chrome and browsed to the website it would wait for the ‘Enter Typing Race’ button to load and it would then click the button using some Javascript I put in the ExecuteJavascript function since I was having issues with the selenium click function.    

Once I join the race and it is starting, I again use some JavaScript to extract the text that I need the bot to type out. 

Once the race starts which I detect by checking the screen for having the images ‘Type this’ or ‘Here’ then I get the bot to type out the text at the speed the user entered in beforehand. 

Once it had finished typing out the text in most cases the website would ask the user to complete the captcha test. So the bot would wait for a button saying to ‘Begin test’ and when it loads up click on it, then it would wait for the image to load then with a bit of JavaScript executed it would get the image data in base64 and store it in a variable for the time being. 

After that it would use python to decode the base64 data and save the captcha as a Png image in the Pictures folder. 

Now that we have the captcha image saved as a file on our desktop, all we had to do was load it into Intellibot using the ReadFromFile image utility and then use an OCR text function to do its best at analyzing the text and then store in in a variable. 

Lastly I made the bot send the text that it analyzed into the captcha textbox and submit it, also I made the bot display a MessageBox showing the text that it analyzed for debug purposes 

Finally the program will end. 

P7 – Implement a control System 

Here are all the functions of my system and there is a video as well of it running: 

Here is it running into an errors on separate occasions which I will talk about later: 

M4 Suggest potential improvements to a control system 

Although my Typeracer Bot does function correctly in most cases it still has flaws which could be worked on as an improvement. 

One of the main flaws that I need to improve is the captcha, the problem with the captcha is there is a random black line in the captcha to make it hard for computers to read using OCR text functions like intellibot does. It is there for this exact purpose described to prevent bots like mine just getting an insane highscore and going straight onto the leaderboard, there needs to be a verification system to prove that it is an actual human typing and not a robot which is why there needs to be a captcha system in place.  

This black line however is not a complete disaster and it’s very common to have things like this in captcha in fact many captchas make the text difficult to read so a computer cannot analyze, in some cases even as a human it is tricky to interpret the text.  

Due to the fact of this method of protection being so popular people have had to resort to image manipulation where they transform images by applying different functions such as Greyscaling, inverting, applying a median and many others to reduce unwanted noise such as the black line in my captcha. When done correctly you can transform the original captcha into a clear and easy to understand captcha for a computer to correctly analyze. 

Therefore, considering that for this type of captcha it is always just a random black line going through the text then I would just have to use some image manipulation techniques to get rid of that random black line and then the OCR text should work perfectly and the bot would pass the test. Unfortunately however, I am not an expert or know anything about image manipulation so I do not have a clue of where I would even begin. I must take the time to study the various image manipulation techniques and learn how to effectively apply the specific ones I need to achieve my desired result. 

Another thing that I need to improve on is the making the bot have less errors. I have noticed through running the bot tens of times that in 9/10 times it work fine as expected but in very few cases it does not operate as expected and it can have errors like shown in the video earlier it did not click the submit button after entering the text from the captcha due to some error that I am not completely sure of I would have to do more research and debugging with Intellibot to figure that out, and in the other screenshot it did not even write anything at all or analyze the text which I need to debug as well but I believe could be due to an error with getting the captcha or saving it. 

D2 – Evaluate the design and performance of a control system. 

The design of my system is almost full proof but could have some adjustments to it to make it full proof.  

The good part of my bot is that it almost always functions properly in all cases this is due to things such as: Using the wait for images function or Wait for create as this actually waits for the actual image/button to load on page before doing anything which is perfect as it will wait the correct amount of time instead of firing off too fast or too late which is exactly what wait for time’s issue is. I also add multiple images to these functions so they have more than one image as backups to try recognize in case they are unable to recognize one or more of them. One part that I should work on is making the Google Chrome Selenium browser resize itself to full because the bot is only capable of recognizing the images when the browser is in full width and height as I took the screenshots of the images in that manner. 

Another part that is good about my design is that I use the ExecuteJavaScript function to execute some javascript in the page which I do because sometimes I have noticed during the production of my bot is that the selenium messes up when trying to select certain elements on the page as it cannot find them sometimes which would be a huge issue for my bot. So I decided to use the ExecuteJavaScript function with some custom JavaScript that I wrote and it always works at finding elements, clicking on them, grabbing the text from containers thus making it 100% reliable for the bot to use instead.  

I feel though that a part I could improve is asking the user the directory where they want to save the captcha image and then store this in a variable which I will refer to throughout the program. This is because I have hardcoded in the directory to be the Pictures folder of the current user. Therefore, this bot will run into an error when run on any other user as it will be unable to located the Pictures folder as the directory for each user will be different depending on the users profile name 

By asking the user to input the preferred directory to save to and storing that in a variable and referencing it this problem will be solved. 

One last problem I would like to solve is Improving the process of grabbing the captcha image. As of now my bot will first ExecuteJavaScript to select the captcha image and grab its data in base64 format, then it will run python code to decode this base64 data and store that as a png image. This can be fairly slow in comparison as to me just copying the image to the clipboard and then using the function ‘GetImage’ from the clipboard which is a lot faster.  

I did do this at the start with the function ‘Click’ which I made right click and then made it recognize where it said ‘Copy Image’ and click on that button. However, the two issues were that one is was fairly slow to recognize the ‘Copy Image’ picture and sometimes it would mess up because the picture did not load fast enough so that’s why I resorted to extracting the base64 data with javascript and decoding and saving it somewhere with python.  

I even tried using the Selenium ‘GetImage’function which is supposed to get the image of the element selected but again this did not work either it just couldn’t get the image for some reason. Unfortunately Javascript does not allow you to directly access the clipboard as this would be a huge security flaw so I was left only really with one option which I could think of which was reliable in nearly all test runs. Perhaps although I could make the mouse right click and use the ‘Move’ function to move the cursor to the general location of where it would say ‘Copy Image’ but after the previous experience of doing something similar to this I feel like I would run into problems of the image loading too slow messing up the whole thing and it might not move the correct amount depending on different screen sizes. 

Another good design part is at the start getting the user to input a speed between 0.02 and 0.09, if they do this successfully it moves the bot onto the rest of the program otherwise it will loop back to asking them to input a correct speed value. It also checks that they do not enter zero as that somehow gets past this barrier.  

The performance of my control system seems to be so far pretty much perfect until it has to solve the captcha: 

In this video you can see me running five back to back tests of my Intellibot and it perfectly executes all of them without any errors. However I may have been lucky in these five tests and I would have to run many more since in very few cases I do run into an error which can be seen in this video and screenshot where for one of them it did not attempt at solving the captcha which I would need to debug to figure out and the other one it didn’t click the submit button. 

The main issue of concern is the captcha as you can see in the video of me running five back to back tests my captcha scores for accuracy was: 49%, 48%, 69%, 64%, 63% giving me an average accuracy score of 59% which is nowhere near good enough to pass the captcha. I believe I would need a minimum captcha score of at least 80-85% to even pass which my bot is nowhere near capable of doing. As explained before the reason why it fails so badly is due to the random black line drawn throughout the captcha which messes up the OCR text function when trying to analyse the text.  

The solution to this problem is to use some image manipulation techniques to get rid of this black line by using different functions such as greyscaling, inverting, applying a median and many others to reduce unwanted noise such as the black line in my captcha. When done correctly you can transform the original captcha into a clear and easy to understand captcha for a computer to correctly analyze. 

Therefore, considering that for this type of captcha it is always just a random black line going through the text then I would just have to use some image manipulation techniques to get rid of that random black line and then the OCR text should work perfectly and the bot would pass the test. Unfortunately however, I am not an expert or know anything about image manipulation so I do not have a clue of where I would even begin. I must take the time to study the various image manipulation techniques and learn how to effectively apply the specific ones I need to achieve my desired result. 

If I managed to successfully do this then I would be able to make my bot click the button to play again so it could infinitely keep playing races scoring top scores and completing the captcha with top accuracy. 

Arduino Blinking LED 

Here is video footage proof and some pictures of a blinking LED project that I made with Arduino 

Here is the code: 

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Google photo

You are commenting using your Google account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s

Create your website with WordPress.com
Get started
%d bloggers like this: