Repository navigation
Other ways to export metrics in hyperparameter tuning jobs #221
Description
Hello,
First of all thank you for implementing Hyperparameter tuning jobs.
This seems like a really useful feature I'd like to use very often.
However, I don't understand why the metrics are collected via logs.
Using regex to search over the logs seems complicated.
The documentation is also confusing without showing an concrete output which will match the shown regex code.
Why are the metrics not gathered via an output json file (in case for custom code)?
E.g. metrics.json which could be created in /op/ml/ouput/
Reacted by ori-n
Activity
[-]Other ways to retrieve metrics in hyperparameter tuning jobs[/-][+]Other ways to export metrics in hyperparameter tuning jobs[/+]dennis-ec commentedon Jun 13, 2018
A little Update: This blog post shows a regex expression as example, which seems more reasonable:
'Validation-accuracy=([0-9\\.]+)'Despite that, the approach to search over logs still seems very inconvenient. So the Feature request for an alternative method still stands.
leopd commentedon Jun 13, 2018
Thanks for the feedback. To explain the "why", one of our goals was to enable hyperparameter tuning on training code that customers don't necessarily understand or control -- e.g. you download a learning algorithm from github or get something from a colleague, and you aren't necessarily comfortable or even able to change the code, but still want to tune it. Regex's solve this problem nicely. As such, it's compatible with nearly every piece of ML code ever written, although it does bring its own set of challenges like crafting the right regex.
I agree the documentation & examples can be better. I've developed some tools to help with testing & debugging the MetricDefinition regexes. I'll look into how we might release such tools publicly.
Merge pull request aws#221 from philipgautier/linear_learner_class_we…