The company mainly uses CLI-based AI tools. It helps with lightweight and fast code work in the terminal environment, and is a great companion for improving work efficiency.
The “Privileges” at Home
At home, I frequently use Codex. Sometimes when I don’t want to sit in front of the computer and hold my hand, I get the fun of putting the AI in control of the computer and watching. I sometimes feel uncomfortable when I come back to the CLI environment after having a tricky repetitive task or experience of AI handling browsing manipulations.
CLI-based AI seems to be stuck at the text and terminal command level, so it’s impossible to see the entire screen and move the mouse in a “intuitive PC control.”
If not, do it yourself: Screenshot + mouse control
In the end, someone who could not sell a well tried to make a simple pipeline.
Screen Capture: Capture the current monitor screen as a screenshot.
Visual analysis: Transfer the captured image to an AI model with Vision functionality.
Input control: performs mouse movements and clicks based on the UI location coordinates analyzed by AI.
It was just a combination of thought-built imagery, but the result was far better than thought.
I know exactly where the button on the screen or the input window is located, and I move my mouse and do the task myself according to the purpose I have instructed. Of course, there is still a bit of a hard part to put it all over, but I still think that if you accumulate the data well, there will be a good result.
How far will it evolve.
Over the age of simply writing code or summarizing sentences, AI has now entered the realm of “agents” where humans understand and execute the interface (GUI) that deals with computers.
This level of control is possible with the small structure of individuals directly interacting, but what if an AI agent that is fully integrated at the OS level appears soon?
Every time you feel the pace of technology development, you feel a bit of admiration and at the same time a bit of fear.
Loading comments.